Sep 19 08:20:30.849788 ip-10-0-131-33 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 08:20:34.284842 ip-10-0-131-33 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:34.284842 ip-10-0-131-33 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 08:20:34.284842 ip-10-0-131-33 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:34.284842 ip-10-0-131-33 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 08:20:34.284842 ip-10-0-131-33 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:34.285820 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.284963    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 08:20:34.288226 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288200    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:34.288226 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288220    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:34.288226 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288226    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:34.288226 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288231    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288235    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288240    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288244    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288248    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288252    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288257    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288261    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288265    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288269    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288273    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288278    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288281    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288285    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288289    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288293    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288298    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288304    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288308    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:34.288517 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288313    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288317    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288321    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288326    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288330    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288334    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288338    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288342    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288347    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288351    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288356    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288360    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288365    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288370    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288375    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288385    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288391    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288395    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288400    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288405    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:34.289368 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288409    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288414    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288419    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288423    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288428    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288432    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288453    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288459    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288463    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288468    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288473    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288478    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288482    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288487    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288491    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288496    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288502    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288506    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288511    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:34.290317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288516    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288521    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288531    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288537    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288542    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288546    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288551    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288556    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288561    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288566    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288570    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288575    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288579    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288584    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288588    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288592    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288596    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288602    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288607    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288612    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:34.291211 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288616    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288621    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288625    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288630    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288635    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288642    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288649    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288654    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288659    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288664    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288669    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288674    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288680    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288685    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288690    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288695    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288700    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.288704    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.289953    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.289963    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:34.292131 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.289968    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.289973    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.289978    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.289982    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.289987    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.289992    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.289997    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290004    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290011    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290016    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290022    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290028    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290033    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290037    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290042    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290048    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290053    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290059    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290063    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:34.293083 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290068    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290072    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290077    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290081    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290086    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290091    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290096    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290101    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290105    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290110    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290114    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290119    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290123    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290128    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290132    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290136    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290141    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290145    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290150    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290154    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:34.293829 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290159    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290163    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290168    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290172    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290176    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290181    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290186    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290190    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290195    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290200    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290208    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290213    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290218    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290223    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290228    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290232    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290237    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290243    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290247    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290252    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:34.294487 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290257    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290262    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290266    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290271    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290275    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290280    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290284    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290289    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290293    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290298    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290305    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290311    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290315    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290319    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290323    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290328    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290332    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290337    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290342    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:34.295078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290347    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290351    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290358    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290364    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290372    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290378    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290383    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290388    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290393    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290398    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290404    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290409    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290414    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290419    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290424    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290429    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290433    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290455    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.290461    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:34.295704 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291107    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291116    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291121    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291126    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291130    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291135    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291140    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291145    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291149    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291154    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291159    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291163    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291169    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291174    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291178    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291183    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291191    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291196    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291200    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291205    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:34.296453 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291208    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291213    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291217    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291221    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291226    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291282    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291287    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291291    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291296    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291300    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291305    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291310    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291314    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291318    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291323    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291327    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291332    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291337    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291341    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291345    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:34.296993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291350    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291355    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291360    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291364    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291368    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291374    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291378    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291384    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291391    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291396    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291401    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291405    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291410    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291414    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291419    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291423    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291427    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291432    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291453    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:34.297634 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291458    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291462    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291467    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291471    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291476    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291480    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291486    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291490    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291495    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291501    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291505    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291509    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291514    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291519    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291523    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291531    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291537    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291544    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291549    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:34.298192 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291557    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291562    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291569    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291575    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291580    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291585    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291590    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291595    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291601    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291607    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291613    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291618    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291623    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291628    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291635    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291640    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291644    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291649    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291656    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:34.298787 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291662    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.291667    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291788    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291847    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291859    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291867    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291874    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291880    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291888    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291895    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291901    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291906    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291912    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291920    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291928    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291935    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291940    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291946    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291951    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.291957    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292026    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292032    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292038    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292043    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 08:20:34.299307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292049    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292056    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292062    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292068    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292074    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292079    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292084    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292089    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292095    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292101    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292108    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292113    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292118    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292124    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292128    2577 flags.go:64] FLAG: --enable-server="true"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292134    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292142    2577 flags.go:64] FLAG: --event-burst="100"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292147    2577 flags.go:64] FLAG: --event-qps="50"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292152    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292158    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292163    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292171    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292176    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292190    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292196    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 19 08:20:34.299948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292202    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292206    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292212    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292217    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292222    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292227    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292232    2577 flags.go:64] FLAG: --feature-gates=""
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292239    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292244    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292250    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292255    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292261    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292267    2577 flags.go:64] FLAG: --help="false"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292272    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292277    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292283    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292288    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292294    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292300    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292305    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292310    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292315    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292320    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 08:20:34.300711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292325    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292331    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292336    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292341    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292346    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292353    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292358    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292365    2577 flags.go:64] FLAG: --lock-file=""
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292371    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292377    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292382    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292392    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292397    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292402    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292407    2577 flags.go:64] FLAG: --logging-format="text"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292412    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292418    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292424    2577 flags.go:64] FLAG: --manifest-url=""
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292429    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292456    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292463    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292470    2577 flags.go:64] FLAG: --max-pods="110"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292476    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292481    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292486    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 08:20:34.301364 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292491    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292496    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292502    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292507    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292520    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292525    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292530    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292535    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292541    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292550    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292555    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292561    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292566    2577 flags.go:64] FLAG: --port="10250"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292571    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292576    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a45792bbf4493fc6"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292584    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292591    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292596    2577 flags.go:64] FLAG: --register-node="true"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292601    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292607    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292612    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292618    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292623    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292629    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292635    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 08:20:34.302113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292640    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292645    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292650    2577 flags.go:64] FLAG: --runonce="false"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292655    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292661    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292667    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292672    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292678    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292683    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292688    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292694    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292699    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292704    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292709    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292714    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292719    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292724    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292729    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292739    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292745    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292750    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292757    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292764    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292769    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292776    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 08:20:34.302899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292782    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292787    2577 flags.go:64] FLAG: --v="2"
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292794    2577 flags.go:64] FLAG: --version="false"
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292801    2577 flags.go:64] FLAG: --vmodule=""
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292808    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.292814    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293011    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293018    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293023    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293028    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293033    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293038    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293043    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293048    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293053    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293058    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293063    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293068    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293075    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293079    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293084    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:34.303609 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293088    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293093    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293099    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293103    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293107    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293112    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293117    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293122    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293128    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293132    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293137    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293143    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293147    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293152    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293156    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293160    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293164    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293169    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293173    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293178    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:34.304165 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293183    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293187    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293192    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293196    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293201    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293205    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293210    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293214    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293219    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293226    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293231    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293235    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293240    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293245    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293250    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293254    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293259    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293263    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293268    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293272    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:34.304727 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293276    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293282    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293286    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293290    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293296    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293300    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293305    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293309    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293313    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293318    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293322    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293327    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293331    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293336    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293340    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293345    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293349    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293353    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293359    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293363    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:34.305254 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293370    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293378    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293382    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293387    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293392    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293396    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293401    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293405    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293409    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293414    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293418    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293426    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293432    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293456    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293461    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293466    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293470    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293476    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:34.305801 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293483    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293489    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293494    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293498    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293503    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.293507    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.294144    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.302556    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.302575    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.302598    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.302608    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302681    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302687    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302691    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302694    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:34.306319 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302697    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302701    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302704    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302707    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302710    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302713    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302716    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302719    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302722    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302725    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302728    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302731    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302734    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302737    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302740    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302743    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302746    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302749    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302753    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302755    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:34.306804 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302758    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302761    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302764    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302767    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302770    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302773    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302777    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302780    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302783    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302786    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302788    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302791    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302794    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302797    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302800    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302803    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302806    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302809    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302811    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302814    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:34.307330 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302817    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302820    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302822    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302825    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302830    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302834    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302837    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302841    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302844    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302848    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302851    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302854    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302857    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302861    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302865    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302868    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302871    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302875    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:34.307878 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302878    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302881    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302886    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302896    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302899    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302902    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302905    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302909    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302912    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302915    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302918    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302921    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302924    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302926    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302929    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302932    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302935    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302938    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302941    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:34.308369 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302944    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302947    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302950    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302953    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302955    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302959    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302962    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302965    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302968    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302971    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302974    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302977    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302981    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302984    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302986    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302989    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302992    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:34.308885 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.302995    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.303000    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303144    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303149    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303152    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303155    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303158    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303162    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303165    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303168    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303172    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303175    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303178    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303181    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303184    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:34.309343 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303187    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303190    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303193    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303196    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303199    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303202    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303205    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303208    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303211    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303214    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303217    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303220    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303224    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303227    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303230    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303233    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303236    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303238    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303241    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303244    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:34.309773 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303247    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303250    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303254    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303257    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303261    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303265    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303269    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303272    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303275    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303278    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303281    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303284    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303287    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303290    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303293    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303296    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303299    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303301    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303304    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303308    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:34.310310 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303310    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303313    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303316    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303319    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303322    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303325    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303328    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303331    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303333    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303336    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303339    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303342    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303345    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303347    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303350    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303353    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303356    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303359    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303361    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303364    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:34.310905 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303367    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303370    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303372    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303375    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303378    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303381    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303383    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303386    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303389    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303392    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303395    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303398    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303403    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303406    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303409    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303413    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303415    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303419    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303421    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:34.311452 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303424    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303427    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303430    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303433    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303448    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303451    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:34.303454    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.303460    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.303594    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.305740    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.306649    2577 server.go:1019] "Starting client certificate rotation"
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.306763    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.306804    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:34.311955 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.309094    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:34.317535 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.310511    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:34.321332 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.321308    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 19 08:20:34.325642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.325621    2577 log.go:25] "Validated CRI v1 image API"
Sep 19 08:20:34.327581 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.327563    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 08:20:34.329985 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.329961    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7d5afc76-65b1-40f1-97a0-5ae736338a04:/dev/nvme0n1p4 f00d2f1c-8a96-4ae9-9a6f-1e1742f4efb3:/dev/nvme0n1p3]
Sep 19 08:20:34.330059 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.329986    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 08:20:34.335524 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.335377    2577 manager.go:216] Machine: {Timestamp:2026-09-19 08:20:34.33408342 +0000 UTC m=+2.015666846 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101796 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea98949bb6ff1af11733d0930860a SystemUUID:ec2ea989-49bb-6ff1-af11-733d0930860a BootID:49816a31-b703-421f-b1bf-4112e225017c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:01:1c:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:01:1c:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:27:3c:6d:5c:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 08:20:34.335524 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.335520    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 08:20:34.335661 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.335652    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 08:20:34.337249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.337221    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 08:20:34.337413 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.337252    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-33.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 08:20:34.338531 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.338520    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 08:20:34.338575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.338535    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 08:20:34.338575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.338544    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:34.338575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.338556    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:34.338660 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.338649    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 08:20:34.343024 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.343009    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:34.343175 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.343164    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 08:20:34.343215 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.343201    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 08:20:34.343256 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.343219    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 08:20:34.343256 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.343237    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 08:20:34.343256 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.343247    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 19 08:20:34.343256 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.343257    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 08:20:34.343726 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.343712    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:34.343799 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.343745    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:34.345828 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.345808    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:34.348320 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.348302    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 08:20:34.350850 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.350835    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 08:20:34.350912 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.350868    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 08:20:34.352037 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352024    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 08:20:34.352093 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352046    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 08:20:34.352093 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352055    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 08:20:34.352093 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352064    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 08:20:34.352093 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352077    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 08:20:34.352093 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352090    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 08:20:34.352235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352112    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 08:20:34.352235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352122    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 08:20:34.352235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352131    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 08:20:34.352235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352139    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 08:20:34.352235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352149    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 08:20:34.352235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352155    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 08:20:34.352235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352179    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 08:20:34.352235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.352184    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 08:20:34.356215 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.356200    2577 server.go:1268] "Started kubelet"
Sep 19 08:20:34.356340 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.356307    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 08:20:34.356397 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.356323    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 08:20:34.356397 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.356383    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 08:20:34.357252 ip-10-0-131-33 systemd[1]: Started Kubernetes Kubelet.
Sep 19 08:20:34.357679 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.357578    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 08:20:34.357832 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.357810    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 08:20:34.357883 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.357834    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-33.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 08:20:34.358005 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.357947    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-33.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 08:20:34.358005 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.357950    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 08:20:34.389392 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.389361    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 08:20:34.389590 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.389404    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:34.390248 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.390208    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:34.390517 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.390489    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 08:20:34.390517 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.390518    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 08:20:34.390756 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.390736    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 08:20:34.390902 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.390887    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 08:20:34.390964 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.390904    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 08:20:34.391498 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.391477    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 08:20:34.391775 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.391753    2577 factory.go:153] Registering CRI-O factory
Sep 19 08:20:34.391848 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.391782    2577 factory.go:223] Registration of the crio container factory successfully
Sep 19 08:20:34.391900 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.391850    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 08:20:34.391900 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.391860    2577 factory.go:55] Registering systemd factory
Sep 19 08:20:34.391900 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.391870    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 19 08:20:34.391900 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.391887    2577 factory.go:103] Registering Raw factory
Sep 19 08:20:34.392085 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.391911    2577 manager.go:1196] Started watching for new ooms in manager
Sep 19 08:20:34.392747 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.392703    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-33.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 19 08:20:34.393752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.393727    2577 manager.go:349] Starting recovery of all containers
Sep 19 08:20:34.403031 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.403005    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 19 08:20:34.403920 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.402982    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-33.ec2.internal.18d6ab1afea674a4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-33.ec2.internal,UID:ip-10-0-131-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-33.ec2.internal,},FirstTimestamp:2026-09-19 08:20:34.35617194 +0000 UTC m=+2.037755368,LastTimestamp:2026-09-19 08:20:34.35617194 +0000 UTC m=+2.037755368,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-33.ec2.internal,}"
Sep 19 08:20:34.405174 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.405155    2577 manager.go:354] Recovery completed
Sep 19 08:20:34.408896 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.408878    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjxz2"
Sep 19 08:20:34.409209 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.409197    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.411580 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.411561    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.411646 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.411596    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.411646 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.411612    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.412265 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.412248    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 08:20:34.412265 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.412260    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.412430 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.412260    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 08:20:34.412430 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.412322    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:34.413051 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.413035    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.413099 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.413062    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.413099 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.413072    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.414671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.414658    2577 policy_none.go:49] "None policy: Start"
Sep 19 08:20:34.414722 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.414675    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 08:20:34.414722 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.414685    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 08:20:34.416822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.416810    2577 policy_none.go:47] "Start"
Sep 19 08:20:34.419541 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.419525    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjxz2"
Sep 19 08:20:34.421439 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.421320    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-33.ec2.internal.18d6ab1b01f3ebf1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-33.ec2.internal,UID:ip-10-0-131-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-33.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-33.ec2.internal,},FirstTimestamp:2026-09-19 08:20:34.411580401 +0000 UTC m=+2.093163828,LastTimestamp:2026-09-19 08:20:34.411580401 +0000 UTC m=+2.093163828,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-33.ec2.internal,}"
Sep 19 08:20:34.423884 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.423818    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-33.ec2.internal.18d6ab1b01f449e0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-33.ec2.internal,UID:ip-10-0-131-33.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-33.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-33.ec2.internal,},FirstTimestamp:2026-09-19 08:20:34.411604448 +0000 UTC m=+2.093187874,LastTimestamp:2026-09-19 08:20:34.411604448 +0000 UTC m=+2.093187874,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-33.ec2.internal,}"
Sep 19 08:20:34.457210 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.457043    2577 manager.go:337] "Starting Device Plugin manager"
Sep 19 08:20:34.466354 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.457250    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 08:20:34.466354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.457281    2577 server.go:87] "Starting device plugin registration server"
Sep 19 08:20:34.466354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.457656    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 08:20:34.466354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.457682    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 08:20:34.466354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.457829    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 08:20:34.466354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.457912    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 08:20:34.466354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.457923    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 08:20:34.466354 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.458546    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 08:20:34.466354 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.458594    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:34.495925 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.495893    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 08:20:34.497220 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.497201    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 08:20:34.497340 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.497247    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 08:20:34.497340 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.497271    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 08:20:34.497340 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.497322    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 08:20:34.499821 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.499794    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:34.558699 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.558596    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.559731 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.559710    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.559822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.559754    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.559822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.559767    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.559822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.559794    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.570154 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.570134    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.570229 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.570157    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-33.ec2.internal\": node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:34.582398 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.582375    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:34.597509 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.597482    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal"]
Sep 19 08:20:34.597609 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.597565    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.598558 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.598534    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.598662 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.598568    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.598662 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.598592    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.601289 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.601272    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.601464 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.601448    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.601524 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.601480    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.602120 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.602103    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.602196 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.602117    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.602196 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.602133    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.602196 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.602147    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.602196 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.602161    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.602196 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.602185    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.605920 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.605896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.606039 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.605927    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:34.606704 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.606689    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:34.606776 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.606718    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:34.606776 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.606729    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:34.634157 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.634136    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-33.ec2.internal\" not found" node="ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.638936 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.638919    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-33.ec2.internal\" not found" node="ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.682838 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.682811    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:34.692934 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.692877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fb7efdd43139469eaac4cb3e953207b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-33.ec2.internal\" (UID: \"9fb7efdd43139469eaac4cb3e953207b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.692996 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.692945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ffa6bc1d275e6dc98ab94a7c0cddfb56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal\" (UID: \"ffa6bc1d275e6dc98ab94a7c0cddfb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.692996 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.692977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffa6bc1d275e6dc98ab94a7c0cddfb56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal\" (UID: \"ffa6bc1d275e6dc98ab94a7c0cddfb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.783000 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.782967    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:34.794187 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.794165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fb7efdd43139469eaac4cb3e953207b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-33.ec2.internal\" (UID: \"9fb7efdd43139469eaac4cb3e953207b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.794240 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.794193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ffa6bc1d275e6dc98ab94a7c0cddfb56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal\" (UID: \"ffa6bc1d275e6dc98ab94a7c0cddfb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.794240 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.794214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffa6bc1d275e6dc98ab94a7c0cddfb56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal\" (UID: \"ffa6bc1d275e6dc98ab94a7c0cddfb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.794323 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.794264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffa6bc1d275e6dc98ab94a7c0cddfb56-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal\" (UID: \"ffa6bc1d275e6dc98ab94a7c0cddfb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.794323 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.794279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fb7efdd43139469eaac4cb3e953207b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-33.ec2.internal\" (UID: \"9fb7efdd43139469eaac4cb3e953207b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.794323 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.794296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ffa6bc1d275e6dc98ab94a7c0cddfb56-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal\" (UID: \"ffa6bc1d275e6dc98ab94a7c0cddfb56\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.883956 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.883924    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:34.938372 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.938329    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.942723 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:34.942699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:34.984977 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:34.984952    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:35.085480 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:35.085452    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:35.186049 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:35.185962    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:35.286431 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:35.286401    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-33.ec2.internal\" not found"
Sep 19 08:20:35.306930 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.306897    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 08:20:35.307053 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.307030    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:35.358492 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.358468    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:35.390424 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.390401    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:35.390590 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.390426    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:35.401277 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.401250    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:35.402714 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.402698    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal"
Sep 19 08:20:35.404230 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.404211    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:35.414669 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.414643    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:35.425827 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.425773    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 08:15:34 +0000 UTC" deadline="2028-04-22 05:55:56.776617093 +0000 UTC"
Sep 19 08:20:35.425945 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.425833    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13941h35m21.350788549s"
Sep 19 08:20:35.427697 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.427677    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6rmhq"
Sep 19 08:20:35.434577 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.434285    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6rmhq"
Sep 19 08:20:35.437884 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.437834    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:35.536884 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:35.536850    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa6bc1d275e6dc98ab94a7c0cddfb56.slice/crio-52536c3be3b8ea720eed73d0c3e719bde723733ef9cc2057f5dd9c228cd18472 WatchSource:0}: Error finding container 52536c3be3b8ea720eed73d0c3e719bde723733ef9cc2057f5dd9c228cd18472: Status 404 returned error can't find the container with id 52536c3be3b8ea720eed73d0c3e719bde723733ef9cc2057f5dd9c228cd18472
Sep 19 08:20:35.537275 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:35.537252    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb7efdd43139469eaac4cb3e953207b.slice/crio-775060784fc2cb8dc1a79f5e0ccba39277161835a01d4a3b5d557b2fb13c06c2 WatchSource:0}: Error finding container 775060784fc2cb8dc1a79f5e0ccba39277161835a01d4a3b5d557b2fb13c06c2: Status 404 returned error can't find the container with id 775060784fc2cb8dc1a79f5e0ccba39277161835a01d4a3b5d557b2fb13c06c2
Sep 19 08:20:35.541795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.541775    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:20:35.965130 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:35.965098    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:36.344332 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.344253    2577 apiserver.go:52] "Watching apiserver"
Sep 19 08:20:36.353476 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.353449    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 08:20:36.353953 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.353922    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8","openshift-dns/node-resolver-ltmhg","openshift-image-registry/node-ca-bzm5f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal","openshift-multus/multus-additional-cni-plugins-6fh57","openshift-network-operator/iptables-alerter-rxjbx","kube-system/konnectivity-agent-vs4dq","openshift-cluster-node-tuning-operator/tuned-znmng","openshift-multus/multus-qr49j","openshift-multus/network-metrics-daemon-kwpz5","openshift-network-diagnostics/network-check-target-p74dm","openshift-ovn-kubernetes/ovnkube-node-6np9d","kube-system/global-pull-secret-syncer-gqvcc"]
Sep 19 08:20:36.363477 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.363455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.365798 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.365755    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:36.365885 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.365855    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:36.367497 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.367263    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n28dm\""
Sep 19 08:20:36.367497 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.367408    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:36.368135 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.368118    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:36.368326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.368308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.368455 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.368335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 08:20:36.370500 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.370201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.371822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.371184    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nj9vm\""
Sep 19 08:20:36.371822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.371191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 08:20:36.371822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.371432    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 08:20:36.372476 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.372443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:36.372568 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.372510    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:36.376043 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.376020    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 08:20:36.376303 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.376289    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.376580 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.376561    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 08:20:36.376943 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.376931    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 08:20:36.377239 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.377172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jkt2p\""
Sep 19 08:20:36.378246 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.378229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.378941 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.378925    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 08:20:36.379030 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.378952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 08:20:36.379503 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.379274    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 08:20:36.379503 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.379289    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 08:20:36.379503 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.379301    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 08:20:36.379647 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.379595    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ztssn\""
Sep 19 08:20:36.380381 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.380344    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:36.380974 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.380958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qnks7\""
Sep 19 08:20:36.381253 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.380981    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:36.382590 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.381573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.385587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.384796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 08:20:36.385587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.385000    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 08:20:36.385587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.385337    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 08:20:36.385587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.385535    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 08:20:36.385795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.385720    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 08:20:36.386084 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.386068    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95pdx\""
Sep 19 08:20:36.386266 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.386252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.387256 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.387013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.388442 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.387833    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 08:20:36.389702 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.389683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:20:36.389951 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.389837    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 08:20:36.390179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.390045    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2ljvz\""
Sep 19 08:20:36.390179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.390087    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 08:20:36.390179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.390119    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 08:20:36.390179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.390120    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jstl7\""
Sep 19 08:20:36.390420 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.390205    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 08:20:36.392010 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.391989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:36.392114 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.392056    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:36.392183 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.392126    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 08:20:36.392400 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.392348    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g9d4h\""
Sep 19 08:20:36.392601 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.392533    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 08:20:36.403345 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:36.403442 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbkcl\" (UniqueName: \"kubernetes.io/projected/d5890e90-fd42-4242-b4ae-1971e659a66e-kube-api-access-cbkcl\") pod \"iptables-alerter-rxjbx\" (UID: \"d5890e90-fd42-4242-b4ae-1971e659a66e\") " pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.403442 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gcb\" (UniqueName: \"kubernetes.io/projected/a791e266-0859-46a0-920f-dcec31e7a326-kube-api-access-w8gcb\") pod \"node-resolver-ltmhg\" (UID: \"a791e266-0859-46a0-920f-dcec31e7a326\") " pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.403442 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-kubernetes\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.403555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-slash\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.403555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:36.403555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4mb5\" (UniqueName: \"kubernetes.io/projected/a0649707-f033-46a5-b1ea-595b2a403f01-kube-api-access-q4mb5\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.403555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-socket-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.403752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5nn9\" (UniqueName: \"kubernetes.io/projected/472a44c8-cd29-40e9-94c8-df7313d11e58-kube-api-access-s5nn9\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.403752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-run\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.403752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-var-lib-kubelet\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.403752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-cni-bin\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.403752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95d065d2-7e01-4e45-83a6-22e2646d868f-kubelet-config\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:36.403752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-etc-selinux\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.403752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-modprobe-d\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-tuned\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-cni-netd\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdb1d57e-d87d-4155-b821-b40189f0e04d-ovnkube-script-lib\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403822    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-system-cni-dir\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-run-netns\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-device-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-host\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.403981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-run-systemd\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d385eb5e-e888-4a6a-9aee-5b23faffb0ec-agent-certs\") pod \"konnectivity-agent-vs4dq\" (UID: \"d385eb5e-e888-4a6a-9aee-5b23faffb0ec\") " pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-registration-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.404080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-systemd-units\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-etc-openvswitch\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-var-lib-cni-multus\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d385eb5e-e888-4a6a-9aee-5b23faffb0ec-konnectivity-ca\") pod \"konnectivity-agent-vs4dq\" (UID: \"d385eb5e-e888-4a6a-9aee-5b23faffb0ec\") " pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5890e90-fd42-4242-b4ae-1971e659a66e-iptables-alerter-script\") pod \"iptables-alerter-rxjbx\" (UID: \"d5890e90-fd42-4242-b4ae-1971e659a66e\") " pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404217    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzfn\" (UniqueName: \"kubernetes.io/projected/c1c234fa-2c7e-43db-9f11-be533024f6a7-kube-api-access-9kzfn\") pod \"node-ca-bzm5f\" (UID: \"c1c234fa-2c7e-43db-9f11-be533024f6a7\") " pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-lib-modules\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-hostroot\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0649707-f033-46a5-b1ea-595b2a403f01-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxszp\" (UniqueName: \"kubernetes.io/projected/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-kube-api-access-hxszp\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-sysctl-conf\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-run-netns\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdb1d57e-d87d-4155-b821-b40189f0e04d-ovn-node-metrics-cert\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0649707-f033-46a5-b1ea-595b2a403f01-system-cni-dir\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-sys-fs\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-node-log\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.404652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzhbp\" (UniqueName: \"kubernetes.io/projected/fdb1d57e-d87d-4155-b821-b40189f0e04d-kube-api-access-wzhbp\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1c234fa-2c7e-43db-9f11-be533024f6a7-host\") pod \"node-ca-bzm5f\" (UID: \"c1c234fa-2c7e-43db-9f11-be533024f6a7\") " pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-run-openvswitch\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-log-socket\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33701976-12b8-41df-9739-1d4cd5cd9e29-tmp\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1c234fa-2c7e-43db-9f11-be533024f6a7-serviceca\") pod \"node-ca-bzm5f\" (UID: \"c1c234fa-2c7e-43db-9f11-be533024f6a7\") " pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-systemd\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-cnibin\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-conf-dir\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.404994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0649707-f033-46a5-b1ea-595b2a403f01-cni-binary-copy\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-sysctl-d\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-var-lib-cni-bin\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-kubelet\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdb1d57e-d87d-4155-b821-b40189f0e04d-env-overrides\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-run-k8s-cni-cncf-io\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405273    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-daemon-config\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.405423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-sysconfig\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdb1d57e-d87d-4155-b821-b40189f0e04d-ovnkube-config\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e3c0738-763f-42cc-b72c-f90bd30e1255-cni-binary-copy\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-var-lib-kubelet\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-run-multus-certs\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvlz\" (UniqueName: \"kubernetes.io/projected/4e3c0738-763f-42cc-b72c-f90bd30e1255-kube-api-access-kzvlz\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95d065d2-7e01-4e45-83a6-22e2646d868f-dbus\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0649707-f033-46a5-b1ea-595b2a403f01-cnibin\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0649707-f033-46a5-b1ea-595b2a403f01-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8r9j\" (UniqueName: \"kubernetes.io/projected/33701976-12b8-41df-9739-1d4cd5cd9e29-kube-api-access-f8r9j\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-var-lib-openvswitch\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-cni-dir\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-etc-kubernetes\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0649707-f033-46a5-b1ea-595b2a403f01-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5890e90-fd42-4242-b4ae-1971e659a66e-host-slash\") pod \"iptables-alerter-rxjbx\" (UID: \"d5890e90-fd42-4242-b4ae-1971e659a66e\") " pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.406208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a791e266-0859-46a0-920f-dcec31e7a326-tmp-dir\") pod \"node-resolver-ltmhg\" (UID: \"a791e266-0859-46a0-920f-dcec31e7a326\") " pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.407016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-sys\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.407016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-run-ovn\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.407016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.405980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.407016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.406002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a791e266-0859-46a0-920f-dcec31e7a326-hosts-file\") pod \"node-resolver-ltmhg\" (UID: \"a791e266-0859-46a0-920f-dcec31e7a326\") " pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.407016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.406037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-socket-dir-parent\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.437334 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.437303    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:35 +0000 UTC" deadline="2028-06-18 18:33:30.296034638 +0000 UTC"
Sep 19 08:20:36.437334 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.437334    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15322h12m53.858705054s"
Sep 19 08:20:36.491273 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.491248    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 08:20:36.502262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.502213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal" event={"ID":"9fb7efdd43139469eaac4cb3e953207b","Type":"ContainerStarted","Data":"775060784fc2cb8dc1a79f5e0ccba39277161835a01d4a3b5d557b2fb13c06c2"}
Sep 19 08:20:36.503441 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.503411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal" event={"ID":"ffa6bc1d275e6dc98ab94a7c0cddfb56","Type":"ContainerStarted","Data":"52536c3be3b8ea720eed73d0c3e719bde723733ef9cc2057f5dd9c228cd18472"}
Sep 19 08:20:36.506263 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33701976-12b8-41df-9739-1d4cd5cd9e29-tmp\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.506381 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1c234fa-2c7e-43db-9f11-be533024f6a7-serviceca\") pod \"node-ca-bzm5f\" (UID: \"c1c234fa-2c7e-43db-9f11-be533024f6a7\") " pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.506381 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-systemd\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.506381 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-cnibin\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.506381 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-conf-dir\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0649707-f033-46a5-b1ea-595b2a403f01-cni-binary-copy\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-sysctl-d\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-systemd\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-var-lib-cni-bin\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-kubelet\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-var-lib-cni-bin\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdb1d57e-d87d-4155-b821-b40189f0e04d-env-overrides\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-run-k8s-cni-cncf-io\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-daemon-config\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-cnibin\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506562    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506598    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-conf-dir\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.506625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-sysconfig\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdb1d57e-d87d-4155-b821-b40189f0e04d-ovnkube-config\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e3c0738-763f-42cc-b72c-f90bd30e1255-cni-binary-copy\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-var-lib-kubelet\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-run-multus-certs\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvlz\" (UniqueName: \"kubernetes.io/projected/4e3c0738-763f-42cc-b72c-f90bd30e1255-kube-api-access-kzvlz\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95d065d2-7e01-4e45-83a6-22e2646d868f-dbus\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0649707-f033-46a5-b1ea-595b2a403f01-cnibin\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506913    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-run-k8s-cni-cncf-io\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0649707-f033-46a5-b1ea-595b2a403f01-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.506967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-kubelet\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8r9j\" (UniqueName: \"kubernetes.io/projected/33701976-12b8-41df-9739-1d4cd5cd9e29-kube-api-access-f8r9j\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-var-lib-openvswitch\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1c234fa-2c7e-43db-9f11-be533024f6a7-serviceca\") pod \"node-ca-bzm5f\" (UID: \"c1c234fa-2c7e-43db-9f11-be533024f6a7\") " pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-cni-dir\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-etc-kubernetes\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.507262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0649707-f033-46a5-b1ea-595b2a403f01-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5890e90-fd42-4242-b4ae-1971e659a66e-host-slash\") pod \"iptables-alerter-rxjbx\" (UID: \"d5890e90-fd42-4242-b4ae-1971e659a66e\") " pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a791e266-0859-46a0-920f-dcec31e7a326-tmp-dir\") pod \"node-resolver-ltmhg\" (UID: \"a791e266-0859-46a0-920f-dcec31e7a326\") " pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-sys\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-run-ovn\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a791e266-0859-46a0-920f-dcec31e7a326-hosts-file\") pod \"node-resolver-ltmhg\" (UID: \"a791e266-0859-46a0-920f-dcec31e7a326\") " pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-socket-dir-parent\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fdb1d57e-d87d-4155-b821-b40189f0e04d-env-overrides\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbkcl\" (UniqueName: \"kubernetes.io/projected/d5890e90-fd42-4242-b4ae-1971e659a66e-kube-api-access-cbkcl\") pod \"iptables-alerter-rxjbx\" (UID: \"d5890e90-fd42-4242-b4ae-1971e659a66e\") " pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gcb\" (UniqueName: \"kubernetes.io/projected/a791e266-0859-46a0-920f-dcec31e7a326-kube-api-access-w8gcb\") pod \"node-resolver-ltmhg\" (UID: \"a791e266-0859-46a0-920f-dcec31e7a326\") " pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-kubernetes\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-slash\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4mb5\" (UniqueName: \"kubernetes.io/projected/a0649707-f033-46a5-b1ea-595b2a403f01-kube-api-access-q4mb5\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-daemon-config\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.507981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-socket-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-socket-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5nn9\" (UniqueName: \"kubernetes.io/projected/472a44c8-cd29-40e9-94c8-df7313d11e58-kube-api-access-s5nn9\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-run\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-var-lib-kubelet\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-cni-bin\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-socket-dir-parent\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-kubernetes\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-run\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0649707-f033-46a5-b1ea-595b2a403f01-cni-binary-copy\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fdb1d57e-d87d-4155-b821-b40189f0e04d-ovnkube-config\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-slash\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0649707-f033-46a5-b1ea-595b2a403f01-cnibin\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-run-ovn\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0649707-f033-46a5-b1ea-595b2a403f01-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-var-lib-kubelet\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-var-lib-kubelet\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.508929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5890e90-fd42-4242-b4ae-1971e659a66e-host-slash\") pod \"iptables-alerter-rxjbx\" (UID: \"d5890e90-fd42-4242-b4ae-1971e659a66e\") " pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-run-multus-certs\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a791e266-0859-46a0-920f-dcec31e7a326-hosts-file\") pod \"node-resolver-ltmhg\" (UID: \"a791e266-0859-46a0-920f-dcec31e7a326\") " pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-sys\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.508427    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95d065d2-7e01-4e45-83a6-22e2646d868f-kubelet-config\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-cni-bin\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.508488    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs podName:5cbb75e1-d58d-41d4-ba14-9e3194e793f8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:37.008466965 +0000 UTC m=+4.690050380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs") pod "network-metrics-daemon-kwpz5" (UID: "5cbb75e1-d58d-41d4-ba14-9e3194e793f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.507855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/95d065d2-7e01-4e45-83a6-22e2646d868f-kubelet-config\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-etc-selinux\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-modprobe-d\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-tuned\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-cni-netd\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-etc-kubernetes\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdb1d57e-d87d-4155-b821-b40189f0e04d-ovnkube-script-lib\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e3c0738-763f-42cc-b72c-f90bd30e1255-cni-binary-copy\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.509770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-system-cni-dir\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-var-lib-openvswitch\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/95d065d2-7e01-4e45-83a6-22e2646d868f-dbus\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-run-netns\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-sysconfig\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a791e266-0859-46a0-920f-dcec31e7a326-tmp-dir\") pod \"node-resolver-ltmhg\" (UID: \"a791e266-0859-46a0-920f-dcec31e7a326\") " pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-device-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-multus-cni-dir\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-run-netns\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-sysctl-d\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-etc-selinux\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-device-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-cni-netd\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-host\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-modprobe-d\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-system-cni-dir\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-run-systemd\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.510933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d385eb5e-e888-4a6a-9aee-5b23faffb0ec-agent-certs\") pod \"konnectivity-agent-vs4dq\" (UID: \"d385eb5e-e888-4a6a-9aee-5b23faffb0ec\") " pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-run-systemd\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-registration-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-systemd-units\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.508961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-etc-openvswitch\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-var-lib-cni-multus\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d385eb5e-e888-4a6a-9aee-5b23faffb0ec-konnectivity-ca\") pod \"konnectivity-agent-vs4dq\" (UID: \"d385eb5e-e888-4a6a-9aee-5b23faffb0ec\") " pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5890e90-fd42-4242-b4ae-1971e659a66e-iptables-alerter-script\") pod \"iptables-alerter-rxjbx\" (UID: \"d5890e90-fd42-4242-b4ae-1971e659a66e\") " pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzfn\" (UniqueName: \"kubernetes.io/projected/c1c234fa-2c7e-43db-9f11-be533024f6a7-kube-api-access-9kzfn\") pod \"node-ca-bzm5f\" (UID: \"c1c234fa-2c7e-43db-9f11-be533024f6a7\") " pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-lib-modules\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-hostroot\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.509165    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0649707-f033-46a5-b1ea-595b2a403f01-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0649707-f033-46a5-b1ea-595b2a403f01-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxszp\" (UniqueName: \"kubernetes.io/projected/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-kube-api-access-hxszp\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.509221    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret podName:95d065d2-7e01-4e45-83a6-22e2646d868f nodeName:}" failed. No retries permitted until 2026-09-19 08:20:37.009206029 +0000 UTC m=+4.690789445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret") pod "global-pull-secret-syncer-gqvcc" (UID: "95d065d2-7e01-4e45-83a6-22e2646d868f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:36.511857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-sysctl-conf\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-run-netns\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdb1d57e-d87d-4155-b821-b40189f0e04d-ovn-node-metrics-cert\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0649707-f033-46a5-b1ea-595b2a403f01-system-cni-dir\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-sys-fs\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-node-log\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzhbp\" (UniqueName: \"kubernetes.io/projected/fdb1d57e-d87d-4155-b821-b40189f0e04d-kube-api-access-wzhbp\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1c234fa-2c7e-43db-9f11-be533024f6a7-host\") pod \"node-ca-bzm5f\" (UID: \"c1c234fa-2c7e-43db-9f11-be533024f6a7\") " pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-run-openvswitch\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-registration-dir\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-log-socket\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fdb1d57e-d87d-4155-b821-b40189f0e04d-ovnkube-script-lib\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-log-socket\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-host\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/472a44c8-cd29-40e9-94c8-df7313d11e58-sys-fs\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-sysctl-conf\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-host-run-netns\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.512664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-node-log\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33701976-12b8-41df-9739-1d4cd5cd9e29-lib-modules\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d5890e90-fd42-4242-b4ae-1971e659a66e-iptables-alerter-script\") pod \"iptables-alerter-rxjbx\" (UID: \"d5890e90-fd42-4242-b4ae-1971e659a66e\") " pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0649707-f033-46a5-b1ea-595b2a403f01-system-cni-dir\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1c234fa-2c7e-43db-9f11-be533024f6a7-host\") pod \"node-ca-bzm5f\" (UID: \"c1c234fa-2c7e-43db-9f11-be533024f6a7\") " pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-run-openvswitch\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-hostroot\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0649707-f033-46a5-b1ea-595b2a403f01-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e3c0738-763f-42cc-b72c-f90bd30e1255-host-var-lib-cni-multus\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.509981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-etc-openvswitch\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.510012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fdb1d57e-d87d-4155-b821-b40189f0e04d-systemd-units\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.510477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d385eb5e-e888-4a6a-9aee-5b23faffb0ec-konnectivity-ca\") pod \"konnectivity-agent-vs4dq\" (UID: \"d385eb5e-e888-4a6a-9aee-5b23faffb0ec\") " pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.511161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33701976-12b8-41df-9739-1d4cd5cd9e29-tmp\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.511178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33701976-12b8-41df-9739-1d4cd5cd9e29-etc-tuned\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.512205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fdb1d57e-d87d-4155-b821-b40189f0e04d-ovn-node-metrics-cert\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.513528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.512562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d385eb5e-e888-4a6a-9aee-5b23faffb0ec-agent-certs\") pod \"konnectivity-agent-vs4dq\" (UID: \"d385eb5e-e888-4a6a-9aee-5b23faffb0ec\") " pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:20:36.517659 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.517627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvlz\" (UniqueName: \"kubernetes.io/projected/4e3c0738-763f-42cc-b72c-f90bd30e1255-kube-api-access-kzvlz\") pod \"multus-qr49j\" (UID: \"4e3c0738-763f-42cc-b72c-f90bd30e1255\") " pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.519186 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.519167    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:36.519296 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.519189    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:36.519296 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.519202    2577 projected.go:196] Error preparing data for projected volume kube-api-access-b496h for pod openshift-network-diagnostics/network-check-target-p74dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:36.519296 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:36.519265    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h podName:ba99474c-b9e8-42c3-8e5d-e899124b24a3 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:37.01924926 +0000 UTC m=+4.700832688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b496h" (UniqueName: "kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h") pod "network-check-target-p74dm" (UID: "ba99474c-b9e8-42c3-8e5d-e899124b24a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:36.519793 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.519768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4mb5\" (UniqueName: \"kubernetes.io/projected/a0649707-f033-46a5-b1ea-595b2a403f01-kube-api-access-q4mb5\") pod \"multus-additional-cni-plugins-6fh57\" (UID: \"a0649707-f033-46a5-b1ea-595b2a403f01\") " pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.520756 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.520732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8r9j\" (UniqueName: \"kubernetes.io/projected/33701976-12b8-41df-9739-1d4cd5cd9e29-kube-api-access-f8r9j\") pod \"tuned-znmng\" (UID: \"33701976-12b8-41df-9739-1d4cd5cd9e29\") " pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.520927 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.520898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5nn9\" (UniqueName: \"kubernetes.io/projected/472a44c8-cd29-40e9-94c8-df7313d11e58-kube-api-access-s5nn9\") pod \"aws-ebs-csi-driver-node-qxzg8\" (UID: \"472a44c8-cd29-40e9-94c8-df7313d11e58\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.521904 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.521878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzhbp\" (UniqueName: \"kubernetes.io/projected/fdb1d57e-d87d-4155-b821-b40189f0e04d-kube-api-access-wzhbp\") pod \"ovnkube-node-6np9d\" (UID: \"fdb1d57e-d87d-4155-b821-b40189f0e04d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.522224 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.522187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gcb\" (UniqueName: \"kubernetes.io/projected/a791e266-0859-46a0-920f-dcec31e7a326-kube-api-access-w8gcb\") pod \"node-resolver-ltmhg\" (UID: \"a791e266-0859-46a0-920f-dcec31e7a326\") " pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.522648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.522615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzfn\" (UniqueName: \"kubernetes.io/projected/c1c234fa-2c7e-43db-9f11-be533024f6a7-kube-api-access-9kzfn\") pod \"node-ca-bzm5f\" (UID: \"c1c234fa-2c7e-43db-9f11-be533024f6a7\") " pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.523180 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.523155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxszp\" (UniqueName: \"kubernetes.io/projected/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-kube-api-access-hxszp\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:36.524354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.524334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbkcl\" (UniqueName: \"kubernetes.io/projected/d5890e90-fd42-4242-b4ae-1971e659a66e-kube-api-access-cbkcl\") pod \"iptables-alerter-rxjbx\" (UID: \"d5890e90-fd42-4242-b4ae-1971e659a66e\") " pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.676949 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.676906    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxjbx"
Sep 19 08:20:36.686771 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.686740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ltmhg"
Sep 19 08:20:36.687389 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.687352    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:36.693031 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.693005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzm5f"
Sep 19 08:20:36.699932 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.699910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6fh57"
Sep 19 08:20:36.707728 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.707703    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-znmng"
Sep 19 08:20:36.714422 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.714400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:20:36.721903 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.721881    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8"
Sep 19 08:20:36.727521 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.727504    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qr49j"
Sep 19 08:20:36.732017 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:36.731999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:20:37.012940 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.012844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:37.012940 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.012916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:37.013179 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:37.013015    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:37.013179 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:37.013053    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:37.013179 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:37.013096    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs podName:5cbb75e1-d58d-41d4-ba14-9e3194e793f8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:38.013075527 +0000 UTC m=+5.694658943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs") pod "network-metrics-daemon-kwpz5" (UID: "5cbb75e1-d58d-41d4-ba14-9e3194e793f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:37.013179 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:37.013117    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret podName:95d065d2-7e01-4e45-83a6-22e2646d868f nodeName:}" failed. No retries permitted until 2026-09-19 08:20:38.013106663 +0000 UTC m=+5.694690075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret") pod "global-pull-secret-syncer-gqvcc" (UID: "95d065d2-7e01-4e45-83a6-22e2646d868f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:37.113705 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.113666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:37.113868 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:37.113829    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:37.113868 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:37.113851    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:37.113868 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:37.113863    2577 projected.go:196] Error preparing data for projected volume kube-api-access-b496h for pod openshift-network-diagnostics/network-check-target-p74dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:37.114031 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:37.113932    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h podName:ba99474c-b9e8-42c3-8e5d-e899124b24a3 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:38.11391276 +0000 UTC m=+5.795496191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b496h" (UniqueName: "kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h") pod "network-check-target-p74dm" (UID: "ba99474c-b9e8-42c3-8e5d-e899124b24a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:37.395987 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:37.395948    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0649707_f033_46a5_b1ea_595b2a403f01.slice/crio-597d272aa4a7e5aee2b09c9ba7ba541f5e26062a92abfa3f5892dba19f4a39c2 WatchSource:0}: Error finding container 597d272aa4a7e5aee2b09c9ba7ba541f5e26062a92abfa3f5892dba19f4a39c2: Status 404 returned error can't find the container with id 597d272aa4a7e5aee2b09c9ba7ba541f5e26062a92abfa3f5892dba19f4a39c2
Sep 19 08:20:37.397340 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:37.397307    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb1d57e_d87d_4155_b821_b40189f0e04d.slice/crio-ac2fdc062fa0e18ab067cbc803c46f7c73ee08f243b0d69e0fd92a431b31ccb1 WatchSource:0}: Error finding container ac2fdc062fa0e18ab067cbc803c46f7c73ee08f243b0d69e0fd92a431b31ccb1: Status 404 returned error can't find the container with id ac2fdc062fa0e18ab067cbc803c46f7c73ee08f243b0d69e0fd92a431b31ccb1
Sep 19 08:20:37.398917 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:37.398867    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda791e266_0859_46a0_920f_dcec31e7a326.slice/crio-9108df97b4e549acdd91602a67bcc07beb1e83890a43ba0c7e04dc7aec8ab47d WatchSource:0}: Error finding container 9108df97b4e549acdd91602a67bcc07beb1e83890a43ba0c7e04dc7aec8ab47d: Status 404 returned error can't find the container with id 9108df97b4e549acdd91602a67bcc07beb1e83890a43ba0c7e04dc7aec8ab47d
Sep 19 08:20:37.403257 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:37.403237    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3c0738_763f_42cc_b72c_f90bd30e1255.slice/crio-ab68f93a2aea9821032752e4683de299967edf1fdd8d0134fe34ee761c4b4ab8 WatchSource:0}: Error finding container ab68f93a2aea9821032752e4683de299967edf1fdd8d0134fe34ee761c4b4ab8: Status 404 returned error can't find the container with id ab68f93a2aea9821032752e4683de299967edf1fdd8d0134fe34ee761c4b4ab8
Sep 19 08:20:37.404195 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:37.404167    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd385eb5e_e888_4a6a_9aee_5b23faffb0ec.slice/crio-42b586e867d1ff0da76eb634661c2cd1cc42afdfcca6fefaf68564a9c41db9e3 WatchSource:0}: Error finding container 42b586e867d1ff0da76eb634661c2cd1cc42afdfcca6fefaf68564a9c41db9e3: Status 404 returned error can't find the container with id 42b586e867d1ff0da76eb634661c2cd1cc42afdfcca6fefaf68564a9c41db9e3
Sep 19 08:20:37.405530 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:37.405512    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33701976_12b8_41df_9739_1d4cd5cd9e29.slice/crio-a7e39ac62c45c89edd4df67979819d525cf3eb40440356adac898fb51b06027a WatchSource:0}: Error finding container a7e39ac62c45c89edd4df67979819d525cf3eb40440356adac898fb51b06027a: Status 404 returned error can't find the container with id a7e39ac62c45c89edd4df67979819d525cf3eb40440356adac898fb51b06027a
Sep 19 08:20:37.406116 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:37.406090    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c234fa_2c7e_43db_9f11_be533024f6a7.slice/crio-7e2a202f64d861e4129d0d652261837b8fcddc543a9a0b112aa43d4e2d0b5eab WatchSource:0}: Error finding container 7e2a202f64d861e4129d0d652261837b8fcddc543a9a0b112aa43d4e2d0b5eab: Status 404 returned error can't find the container with id 7e2a202f64d861e4129d0d652261837b8fcddc543a9a0b112aa43d4e2d0b5eab
Sep 19 08:20:37.407875 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:37.407726    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5890e90_fd42_4242_b4ae_1971e659a66e.slice/crio-b1d9283d4b0422ba2073839cf0dd636c4ea047e581f3a09137df180f5931c65b WatchSource:0}: Error finding container b1d9283d4b0422ba2073839cf0dd636c4ea047e581f3a09137df180f5931c65b: Status 404 returned error can't find the container with id b1d9283d4b0422ba2073839cf0dd636c4ea047e581f3a09137df180f5931c65b
Sep 19 08:20:37.408956 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:20:37.408925    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472a44c8_cd29_40e9_94c8_df7313d11e58.slice/crio-3c67984ff1de77b8b7477f6fc7b10848ea6ea6aeb82682aa0ff3b512d90b3c12 WatchSource:0}: Error finding container 3c67984ff1de77b8b7477f6fc7b10848ea6ea6aeb82682aa0ff3b512d90b3c12: Status 404 returned error can't find the container with id 3c67984ff1de77b8b7477f6fc7b10848ea6ea6aeb82682aa0ff3b512d90b3c12
Sep 19 08:20:37.438256 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.438201    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:35 +0000 UTC" deadline="2028-03-16 13:32:57.025248581 +0000 UTC"
Sep 19 08:20:37.438256 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.438254    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13061h12m19.586998341s"
Sep 19 08:20:37.506942 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.506911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxjbx" event={"ID":"d5890e90-fd42-4242-b4ae-1971e659a66e","Type":"ContainerStarted","Data":"b1d9283d4b0422ba2073839cf0dd636c4ea047e581f3a09137df180f5931c65b"}
Sep 19 08:20:37.508089 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.508063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-znmng" event={"ID":"33701976-12b8-41df-9739-1d4cd5cd9e29","Type":"ContainerStarted","Data":"a7e39ac62c45c89edd4df67979819d525cf3eb40440356adac898fb51b06027a"}
Sep 19 08:20:37.509153 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.509126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ltmhg" event={"ID":"a791e266-0859-46a0-920f-dcec31e7a326","Type":"ContainerStarted","Data":"9108df97b4e549acdd91602a67bcc07beb1e83890a43ba0c7e04dc7aec8ab47d"}
Sep 19 08:20:37.510421 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.510393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerStarted","Data":"ac2fdc062fa0e18ab067cbc803c46f7c73ee08f243b0d69e0fd92a431b31ccb1"}
Sep 19 08:20:37.511640 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.511617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8" event={"ID":"472a44c8-cd29-40e9-94c8-df7313d11e58","Type":"ContainerStarted","Data":"3c67984ff1de77b8b7477f6fc7b10848ea6ea6aeb82682aa0ff3b512d90b3c12"}
Sep 19 08:20:37.512779 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.512755    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vs4dq" event={"ID":"d385eb5e-e888-4a6a-9aee-5b23faffb0ec","Type":"ContainerStarted","Data":"42b586e867d1ff0da76eb634661c2cd1cc42afdfcca6fefaf68564a9c41db9e3"}
Sep 19 08:20:37.513906 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.513881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qr49j" event={"ID":"4e3c0738-763f-42cc-b72c-f90bd30e1255","Type":"ContainerStarted","Data":"ab68f93a2aea9821032752e4683de299967edf1fdd8d0134fe34ee761c4b4ab8"}
Sep 19 08:20:37.514966 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.514944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fh57" event={"ID":"a0649707-f033-46a5-b1ea-595b2a403f01","Type":"ContainerStarted","Data":"597d272aa4a7e5aee2b09c9ba7ba541f5e26062a92abfa3f5892dba19f4a39c2"}
Sep 19 08:20:37.516882 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.516858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal" event={"ID":"9fb7efdd43139469eaac4cb3e953207b","Type":"ContainerStarted","Data":"0ea04b4e8e4114ffafcfabf5a035257531ad23b7d4f085318ed91e04ff06422e"}
Sep 19 08:20:37.517940 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:37.517915    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzm5f" event={"ID":"c1c234fa-2c7e-43db-9f11-be533024f6a7","Type":"ContainerStarted","Data":"7e2a202f64d861e4129d0d652261837b8fcddc543a9a0b112aa43d4e2d0b5eab"}
Sep 19 08:20:38.020214 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:38.019897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:38.020214 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:38.020195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:38.020214 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.020195    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:38.020513 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.020281    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs podName:5cbb75e1-d58d-41d4-ba14-9e3194e793f8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:40.02025843 +0000 UTC m=+7.701841862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs") pod "network-metrics-daemon-kwpz5" (UID: "5cbb75e1-d58d-41d4-ba14-9e3194e793f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:38.020513 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.020325    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:38.020513 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.020394    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret podName:95d065d2-7e01-4e45-83a6-22e2646d868f nodeName:}" failed. No retries permitted until 2026-09-19 08:20:40.020378052 +0000 UTC m=+7.701961467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret") pod "global-pull-secret-syncer-gqvcc" (UID: "95d065d2-7e01-4e45-83a6-22e2646d868f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:38.121203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:38.121143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:38.121458 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.121434    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:38.121545 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.121468    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:38.121545 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.121485    2577 projected.go:196] Error preparing data for projected volume kube-api-access-b496h for pod openshift-network-diagnostics/network-check-target-p74dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:38.121545 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.121544    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h podName:ba99474c-b9e8-42c3-8e5d-e899124b24a3 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:40.121524139 +0000 UTC m=+7.803107553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b496h" (UniqueName: "kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h") pod "network-check-target-p74dm" (UID: "ba99474c-b9e8-42c3-8e5d-e899124b24a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:38.500777 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:38.500743    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:38.501377 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.500898    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:38.501377 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:38.501309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:38.501500 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.501417    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:38.501500 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:38.501488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:38.501601 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:38.501555    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:38.533889 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:38.533853    2577 generic.go:356] "Generic (PLEG): container finished" podID="ffa6bc1d275e6dc98ab94a7c0cddfb56" containerID="1a20fb2841aa21c58da3eeb240d0b17d5a7b2cae6068764b0a872c00c0bbd692" exitCode=0
Sep 19 08:20:38.534950 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:38.534921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal" event={"ID":"ffa6bc1d275e6dc98ab94a7c0cddfb56","Type":"ContainerDied","Data":"1a20fb2841aa21c58da3eeb240d0b17d5a7b2cae6068764b0a872c00c0bbd692"}
Sep 19 08:20:38.558507 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:38.558443    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-33.ec2.internal" podStartSLOduration=3.558424011 podStartE2EDuration="3.558424011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:37.533536804 +0000 UTC m=+5.215120238" watchObservedRunningTime="2026-09-19 08:20:38.558424011 +0000 UTC m=+6.240007444"
Sep 19 08:20:39.557944 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:39.557759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal" event={"ID":"ffa6bc1d275e6dc98ab94a7c0cddfb56","Type":"ContainerStarted","Data":"51a9b47d15537379bc9c5d57f175405df430d752e6ef7da3890e1bd0e0ad6637"}
Sep 19 08:20:39.574563 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:39.574508    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-33.ec2.internal" podStartSLOduration=4.574486882 podStartE2EDuration="4.574486882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:39.57348379 +0000 UTC m=+7.255067224" watchObservedRunningTime="2026-09-19 08:20:39.574486882 +0000 UTC m=+7.256070324"
Sep 19 08:20:40.037889 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:40.037850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:40.038060 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:40.037909    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:40.038120 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.038072    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:40.038165 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.038135    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret podName:95d065d2-7e01-4e45-83a6-22e2646d868f nodeName:}" failed. No retries permitted until 2026-09-19 08:20:44.038117147 +0000 UTC m=+11.719700577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret") pod "global-pull-secret-syncer-gqvcc" (UID: "95d065d2-7e01-4e45-83a6-22e2646d868f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:40.038566 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.038545    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:40.038694 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.038602    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs podName:5cbb75e1-d58d-41d4-ba14-9e3194e793f8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:44.038587917 +0000 UTC m=+11.720171342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs") pod "network-metrics-daemon-kwpz5" (UID: "5cbb75e1-d58d-41d4-ba14-9e3194e793f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:40.138329 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:40.138282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:40.138535 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.138488    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:40.138535 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.138510    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:40.138535 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.138521    2577 projected.go:196] Error preparing data for projected volume kube-api-access-b496h for pod openshift-network-diagnostics/network-check-target-p74dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:40.138685 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.138592    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h podName:ba99474c-b9e8-42c3-8e5d-e899124b24a3 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:44.138573161 +0000 UTC m=+11.820156588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b496h" (UniqueName: "kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h") pod "network-check-target-p74dm" (UID: "ba99474c-b9e8-42c3-8e5d-e899124b24a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:40.498169 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:40.498130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:40.498408 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.498274    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:40.498831 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:40.498804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:40.498973 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.498904    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:40.499059 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:40.498974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:40.499059 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:40.499039    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:42.498556 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:42.498467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:42.498952 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:42.498493    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:42.498952 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:42.498630    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:42.498952 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:42.498696    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:42.498952 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:42.498757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:42.498952 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:42.498834    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:44.073203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:44.073163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:44.073617 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:44.073230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:44.073617 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.073349    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:44.073617 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.073427    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret podName:95d065d2-7e01-4e45-83a6-22e2646d868f nodeName:}" failed. No retries permitted until 2026-09-19 08:20:52.073408863 +0000 UTC m=+19.754992278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret") pod "global-pull-secret-syncer-gqvcc" (UID: "95d065d2-7e01-4e45-83a6-22e2646d868f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:44.073721 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.073620    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:44.073721 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.073663    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs podName:5cbb75e1-d58d-41d4-ba14-9e3194e793f8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:52.073649851 +0000 UTC m=+19.755233267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs") pod "network-metrics-daemon-kwpz5" (UID: "5cbb75e1-d58d-41d4-ba14-9e3194e793f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:44.174087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:44.174048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:44.174260 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.174192    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:44.174260 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.174216    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:44.174260 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.174229    2577 projected.go:196] Error preparing data for projected volume kube-api-access-b496h for pod openshift-network-diagnostics/network-check-target-p74dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:44.174542 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.174280    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h podName:ba99474c-b9e8-42c3-8e5d-e899124b24a3 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:52.174265675 +0000 UTC m=+19.855849105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b496h" (UniqueName: "kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h") pod "network-check-target-p74dm" (UID: "ba99474c-b9e8-42c3-8e5d-e899124b24a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:44.499413 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:44.498935    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:44.499413 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:44.499038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:44.499413 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.499051    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:44.499413 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.499172    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:44.499413 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:44.499210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:44.499413 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:44.499320    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:46.497962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:46.497920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:46.497962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:46.497949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:46.498500 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:46.497922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:46.498500 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:46.498034    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:46.498500 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:46.498110    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:46.498500 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:46.498207    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:48.498489 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:48.498449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:48.498924 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:48.498449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:48.498924 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:48.498596    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:48.498924 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:48.498449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:48.498924 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:48.498697    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:48.498924 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:48.498784    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:50.497966 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:50.497929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:50.498445 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:50.497974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:50.498445 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:50.497937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:50.498445 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:50.498115    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:50.498445 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:50.498197    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:50.498445 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:50.498305    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:52.135739 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:52.135705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:52.136173 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:52.135745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:52.136173 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.135862    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:52.136173 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.135917    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret podName:95d065d2-7e01-4e45-83a6-22e2646d868f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.135902366 +0000 UTC m=+35.817485776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret") pod "global-pull-secret-syncer-gqvcc" (UID: "95d065d2-7e01-4e45-83a6-22e2646d868f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:52.136173 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.135862    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:52.136173 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.135981    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs podName:5cbb75e1-d58d-41d4-ba14-9e3194e793f8 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.135969878 +0000 UTC m=+35.817553307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs") pod "network-metrics-daemon-kwpz5" (UID: "5cbb75e1-d58d-41d4-ba14-9e3194e793f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:52.236715 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:52.236678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:52.236907 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.236863    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:52.236907 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.236889    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:52.236907 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.236901    2577 projected.go:196] Error preparing data for projected volume kube-api-access-b496h for pod openshift-network-diagnostics/network-check-target-p74dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:52.237052 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.236969    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h podName:ba99474c-b9e8-42c3-8e5d-e899124b24a3 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.236951728 +0000 UTC m=+35.918535155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b496h" (UniqueName: "kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h") pod "network-check-target-p74dm" (UID: "ba99474c-b9e8-42c3-8e5d-e899124b24a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:52.498193 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:52.498113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:52.498389 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:52.498117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:52.498389 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.498246    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:52.498389 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:52.498119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:52.498389 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.498318    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:52.498586 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:52.498453    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:54.498434 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:54.498400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:54.498805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:54.498486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:54.498805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:54.498518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:54.498805 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:54.498485    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:54.498805 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:54.498572    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:54.498805 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:54.498615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:55.593000 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.592968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8" event={"ID":"472a44c8-cd29-40e9-94c8-df7313d11e58","Type":"ContainerStarted","Data":"d0619dc9063c287b0e455b8a46f33655e5db69b8df2da415774a2c99e0682dca"}
Sep 19 08:20:55.594671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.594639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vs4dq" event={"ID":"d385eb5e-e888-4a6a-9aee-5b23faffb0ec","Type":"ContainerStarted","Data":"3df6ea2c7b5fb66353f375a8b7b439b0365eaac9acdb3cab8ca6a73c43e957f9"}
Sep 19 08:20:55.596559 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.596532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qr49j" event={"ID":"4e3c0738-763f-42cc-b72c-f90bd30e1255","Type":"ContainerStarted","Data":"b2613404e2183f030e72d58a668e90ccab496f562b5745727e5bd149705d9015"}
Sep 19 08:20:55.598341 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.598315    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0649707-f033-46a5-b1ea-595b2a403f01" containerID="f655aed2697c86a6f9975f66dd20c13a26655a1c2574b7fc0e92a8d733812172" exitCode=0
Sep 19 08:20:55.598475 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.598405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fh57" event={"ID":"a0649707-f033-46a5-b1ea-595b2a403f01","Type":"ContainerDied","Data":"f655aed2697c86a6f9975f66dd20c13a26655a1c2574b7fc0e92a8d733812172"}
Sep 19 08:20:55.600041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.600018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzm5f" event={"ID":"c1c234fa-2c7e-43db-9f11-be533024f6a7","Type":"ContainerStarted","Data":"2cb1d0e250da6d5003ff15c8f54afa4f44ee9a5398a5ac511a773d50ce81e8e2"}
Sep 19 08:20:55.601925 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.601902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-znmng" event={"ID":"33701976-12b8-41df-9739-1d4cd5cd9e29","Type":"ContainerStarted","Data":"19bf55984ae65021ab86494dcc83235f68bbee622f9b369d699917e3d1d0c23e"}
Sep 19 08:20:55.603908 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.603882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ltmhg" event={"ID":"a791e266-0859-46a0-920f-dcec31e7a326","Type":"ContainerStarted","Data":"8c1a367dd7ffd3e8cc87b68b227c92a8b008376dcda5742dba323e2810fb0639"}
Sep 19 08:20:55.606805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.606744    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:20:55.607545 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.607518    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdb1d57e-d87d-4155-b821-b40189f0e04d" containerID="2515c841200341002c1425fbc5af97e44caa61c500a9dbb4ae99d2dced73eb68" exitCode=1
Sep 19 08:20:55.607641 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.607556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerStarted","Data":"549c436ca2c8604c3927b844a8ca93e4afa308dd370696366d110790b30f2fad"}
Sep 19 08:20:55.607641 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.607618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerDied","Data":"2515c841200341002c1425fbc5af97e44caa61c500a9dbb4ae99d2dced73eb68"}
Sep 19 08:20:55.607753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.607664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerStarted","Data":"d4db3cc30f927d0479b8699781f751a3aa6e8448b85195f9042d8f336220d1ef"}
Sep 19 08:20:55.611114 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.611037    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vs4dq" podStartSLOduration=11.605236854 podStartE2EDuration="21.611025846s" totalImagesPullingTime="10.005788992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.610521364 +0000 UTC m=+23.292104801" watchObservedRunningTime="2026-09-19 08:20:55.611025846 +0000 UTC m=+23.292609278"
Sep 19 08:20:55.645533 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.645330    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-znmng" podStartSLOduration=4.175233249 podStartE2EDuration="21.645307019s" totalImagesPullingTime="17.47007377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.627221959 +0000 UTC m=+23.308805394" watchObservedRunningTime="2026-09-19 08:20:55.645307019 +0000 UTC m=+23.326890453"
Sep 19 08:20:55.645645 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.645602    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qr49j" podStartSLOduration=4.128194461 podStartE2EDuration="21.645595493s" totalImagesPullingTime="17.517401032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.645124958 +0000 UTC m=+23.326708390" watchObservedRunningTime="2026-09-19 08:20:55.645595493 +0000 UTC m=+23.327178927"
Sep 19 08:20:55.660549 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.660489    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ltmhg" podStartSLOduration=4.2157640690000004 podStartE2EDuration="21.660468319s" totalImagesPullingTime="17.44470425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.66034735 +0000 UTC m=+23.341930784" watchObservedRunningTime="2026-09-19 08:20:55.660468319 +0000 UTC m=+23.342051753"
Sep 19 08:20:55.677046 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:55.676990    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bzm5f" podStartSLOduration=4.209797876 podStartE2EDuration="21.676969045s" totalImagesPullingTime="17.467171169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.676412747 +0000 UTC m=+23.357996319" watchObservedRunningTime="2026-09-19 08:20:55.676969045 +0000 UTC m=+23.358552479"
Sep 19 08:20:56.134650 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.134608    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 08:20:56.471818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.471681    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T08:20:56.134629808Z","UUID":"9deb2d69-69aa-496b-8999-f151cf87c4e9","Handler":null,"Name":"","Endpoint":""}
Sep 19 08:20:56.473404 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.473379    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 08:20:56.473517 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.473417    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 08:20:56.498738 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.498707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:56.498920 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:56.498837    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:56.499173 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.499153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:56.499173 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.499166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:56.499284 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:56.499262    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:56.499467 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:56.499444    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:56.612753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.612715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxjbx" event={"ID":"d5890e90-fd42-4242-b4ae-1971e659a66e","Type":"ContainerStarted","Data":"5be3a506999400683c7c1b3f28b065fede30ff121e9fd42298aa0af1e7ba6cbf"}
Sep 19 08:20:56.617053 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.617028    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:20:56.617742 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.617553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerStarted","Data":"a95c84e3cffed1f4f6fda1596f601897d480aca554b36a8539b197b8900929d8"}
Sep 19 08:20:56.617742 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.617638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerStarted","Data":"bd0f600d2c9acbeb348c8f5b1f379c92fdaa981f81fe3fae40b7c86adb610f08"}
Sep 19 08:20:56.617742 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.617687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerStarted","Data":"2b409ac5b6e8f6a6ffc021b78b7cd59b9925cbcdde9a8805e9e607599232f5ee"}
Sep 19 08:20:56.620167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.620139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8" event={"ID":"472a44c8-cd29-40e9-94c8-df7313d11e58","Type":"ContainerStarted","Data":"86ca3e5a1909566b9cb9611e4baa3663c6b4d9357b12dd7952804ca4f4f1f999"}
Sep 19 08:20:56.682126 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:56.682093    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:20:57.624257 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:57.624217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8" event={"ID":"472a44c8-cd29-40e9-94c8-df7313d11e58","Type":"ContainerStarted","Data":"3371f27177bed490fbaee146c0920c58754f4518e6674b1e444e1d2e41bb3b8d"}
Sep 19 08:20:57.641807 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:57.641753    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rxjbx" podStartSLOduration=6.175446044 podStartE2EDuration="23.641737737s" totalImagesPullingTime="17.466291693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:56.63000393 +0000 UTC m=+24.311587364" watchObservedRunningTime="2026-09-19 08:20:57.641737737 +0000 UTC m=+25.323321168"
Sep 19 08:20:57.642313 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:57.642267    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxzg8" podStartSLOduration=4.334222992 podStartE2EDuration="23.642255883s" totalImagesPullingTime="19.308032891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:57.641642732 +0000 UTC m=+25.323226165" watchObservedRunningTime="2026-09-19 08:20:57.642255883 +0000 UTC m=+25.323839316"
Sep 19 08:20:58.498253 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:58.498214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:20:58.498460 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:58.498220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:20:58.498460 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:58.498379    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:20:58.498460 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:58.498232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:20:58.498460 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:58.498442    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:20:58.498656 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:20:58.498523    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:20:58.629635 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:58.629605    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:20:58.630066 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:20:58.630011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerStarted","Data":"3a507ecc03d5955414e7bd8bc558ccd95d30da753b4e5b0e8c5f486cd4acf3f4"}
Sep 19 08:21:00.177350 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.177175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:21:00.177834 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.177796    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:21:00.498008 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.497973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:00.498204 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.497984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:00.498204 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:00.498067    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:21:00.498327 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:00.498201    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:21:00.498327 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.498247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:00.498460 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:00.498341    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:21:00.637765 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.637737    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:21:00.638117 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.638091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerStarted","Data":"d324cfdb6562d05012c91471c1693394ee597d286bfe06fee7f04f22334a2aea"}
Sep 19 08:21:00.638319 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.638298    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:21:00.638621 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.638514    2577 scope.go:117] "RemoveContainer" containerID="2515c841200341002c1425fbc5af97e44caa61c500a9dbb4ae99d2dced73eb68"
Sep 19 08:21:00.640209 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.640185    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0649707-f033-46a5-b1ea-595b2a403f01" containerID="cc0826689e4c173b3545dea68dfe3fb98584c658e590abda9344a80363aba312" exitCode=0
Sep 19 08:21:00.640334 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.640311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fh57" event={"ID":"a0649707-f033-46a5-b1ea-595b2a403f01","Type":"ContainerDied","Data":"cc0826689e4c173b3545dea68dfe3fb98584c658e590abda9344a80363aba312"}
Sep 19 08:21:00.641219 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.641119    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vs4dq"
Sep 19 08:21:00.655045 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:00.655011    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:21:01.646872 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:01.646723    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:21:01.647327 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:01.647294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" event={"ID":"fdb1d57e-d87d-4155-b821-b40189f0e04d","Type":"ContainerStarted","Data":"fb6df3cab189e3fbce0c2b751be618c578057520224d02326441cfbc599989c5"}
Sep 19 08:21:01.647394 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:01.647358    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:01.647583 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:01.647567    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:21:01.662274 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:01.662253    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:21:01.681463 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:01.681412    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d" podStartSLOduration=10.193615902 podStartE2EDuration="27.681396636s" totalImagesPullingTime="17.487780734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:01.68043533 +0000 UTC m=+29.362018762" watchObservedRunningTime="2026-09-19 08:21:01.681396636 +0000 UTC m=+29.362980046"
Sep 19 08:21:02.001998 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.001963    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqvcc"]
Sep 19 08:21:02.002232 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.002214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:02.002401 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:02.002335    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:21:02.002713 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.002696    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p74dm"]
Sep 19 08:21:02.002802 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.002773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:02.002866 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:02.002834    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:21:02.005903 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.005881    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kwpz5"]
Sep 19 08:21:02.006008 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.005985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:02.006072 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:02.006056    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:21:02.651060 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.651024    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0649707-f033-46a5-b1ea-595b2a403f01" containerID="6d523029a92ec4f42e277496046f02fd044f2be4866afce27e61151c4e3e3ee8" exitCode=0
Sep 19 08:21:02.651501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.651092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fh57" event={"ID":"a0649707-f033-46a5-b1ea-595b2a403f01","Type":"ContainerDied","Data":"6d523029a92ec4f42e277496046f02fd044f2be4866afce27e61151c4e3e3ee8"}
Sep 19 08:21:02.651501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.651204    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:02.908962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:02.908873    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:21:03.498006 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:03.497967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:03.498167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:03.498015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:03.498167 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:03.498105    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:21:03.498167 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:03.498149    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:21:03.498341 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:03.498209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:03.498341 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:03.498319    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:21:04.657877 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:04.657847    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0649707-f033-46a5-b1ea-595b2a403f01" containerID="b60ae41efdbb5a9af0ce179f74ab6c4a56cff00d89790b54272e37bf1713f85b" exitCode=0
Sep 19 08:21:04.658278 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:04.657924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fh57" event={"ID":"a0649707-f033-46a5-b1ea-595b2a403f01","Type":"ContainerDied","Data":"b60ae41efdbb5a9af0ce179f74ab6c4a56cff00d89790b54272e37bf1713f85b"}
Sep 19 08:21:05.498773 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:05.498546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:05.498938 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:05.498547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:05.498938 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:05.498852    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:21:05.499045 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:05.498959    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:21:05.499045 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:05.498589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:05.499154 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:05.499074    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:21:07.498284 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.498243    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:07.498284 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.498270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:07.498876 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.498279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:07.498876 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:07.498423    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqvcc" podUID="95d065d2-7e01-4e45-83a6-22e2646d868f"
Sep 19 08:21:07.498876 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:07.498522    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p74dm" podUID="ba99474c-b9e8-42c3-8e5d-e899124b24a3"
Sep 19 08:21:07.498876 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:07.498610    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kwpz5" podUID="5cbb75e1-d58d-41d4-ba14-9e3194e793f8"
Sep 19 08:21:07.772219 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.772144    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-33.ec2.internal" event="NodeReady"
Sep 19 08:21:07.772394 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.772293    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 08:21:07.829796 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.829757    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kdt58"]
Sep 19 08:21:07.837642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.837613    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5ld78"]
Sep 19 08:21:07.837783 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.837740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:07.840906 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.840882    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wm7fh\""
Sep 19 08:21:07.841182 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.841161    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 08:21:07.841378 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.841345    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 08:21:07.843339 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.843317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5ld78"
Sep 19 08:21:07.846522 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.846452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 08:21:07.846726 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.846707    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 08:21:07.846822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.846743    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vj75p\""
Sep 19 08:21:07.846822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.846762    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 08:21:07.846822 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.846770    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kdt58"]
Sep 19 08:21:07.854893 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.854871    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5ld78"]
Sep 19 08:21:07.955132 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.955094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8d7r\" (UniqueName: \"kubernetes.io/projected/28834fd2-afab-47de-a085-52668ee84b6f-kube-api-access-w8d7r\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:07.955314 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.955144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28834fd2-afab-47de-a085-52668ee84b6f-tmp-dir\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:07.955314 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.955164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be02c1ed-a740-42d1-be31-3290295ef972-cert\") pod \"ingress-canary-5ld78\" (UID: \"be02c1ed-a740-42d1-be31-3290295ef972\") " pod="openshift-ingress-canary/ingress-canary-5ld78"
Sep 19 08:21:07.955314 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.955211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f2hz\" (UniqueName: \"kubernetes.io/projected/be02c1ed-a740-42d1-be31-3290295ef972-kube-api-access-9f2hz\") pod \"ingress-canary-5ld78\" (UID: \"be02c1ed-a740-42d1-be31-3290295ef972\") " pod="openshift-ingress-canary/ingress-canary-5ld78"
Sep 19 08:21:07.955490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.955341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:07.955490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:07.955393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/28834fd2-afab-47de-a085-52668ee84b6f-config-volume\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:08.056556 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.056467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:08.056556 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.056528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/28834fd2-afab-47de-a085-52668ee84b6f-config-volume\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:08.056782 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.056558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8d7r\" (UniqueName: \"kubernetes.io/projected/28834fd2-afab-47de-a085-52668ee84b6f-kube-api-access-w8d7r\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:08.056782 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.056585    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28834fd2-afab-47de-a085-52668ee84b6f-tmp-dir\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:08.056782 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.056605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be02c1ed-a740-42d1-be31-3290295ef972-cert\") pod \"ingress-canary-5ld78\" (UID: \"be02c1ed-a740-42d1-be31-3290295ef972\") " pod="openshift-ingress-canary/ingress-canary-5ld78"
Sep 19 08:21:08.056782 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.056624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f2hz\" (UniqueName: \"kubernetes.io/projected/be02c1ed-a740-42d1-be31-3290295ef972-kube-api-access-9f2hz\") pod \"ingress-canary-5ld78\" (UID: \"be02c1ed-a740-42d1-be31-3290295ef972\") " pod="openshift-ingress-canary/ingress-canary-5ld78"
Sep 19 08:21:08.056782 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.056649    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:08.056782 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.056728    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls podName:28834fd2-afab-47de-a085-52668ee84b6f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.55670663 +0000 UTC m=+36.238290057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls") pod "dns-default-kdt58" (UID: "28834fd2-afab-47de-a085-52668ee84b6f") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:08.057337 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.057316    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28834fd2-afab-47de-a085-52668ee84b6f-tmp-dir\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:08.057483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.057463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/28834fd2-afab-47de-a085-52668ee84b6f-config-volume\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:08.061586 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.061564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be02c1ed-a740-42d1-be31-3290295ef972-cert\") pod \"ingress-canary-5ld78\" (UID: \"be02c1ed-a740-42d1-be31-3290295ef972\") " pod="openshift-ingress-canary/ingress-canary-5ld78"
Sep 19 08:21:08.068040 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.068008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f2hz\" (UniqueName: \"kubernetes.io/projected/be02c1ed-a740-42d1-be31-3290295ef972-kube-api-access-9f2hz\") pod \"ingress-canary-5ld78\" (UID: \"be02c1ed-a740-42d1-be31-3290295ef972\") " pod="openshift-ingress-canary/ingress-canary-5ld78"
Sep 19 08:21:08.068671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.068651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8d7r\" (UniqueName: \"kubernetes.io/projected/28834fd2-afab-47de-a085-52668ee84b6f-kube-api-access-w8d7r\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:08.156844 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.156806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5ld78"
Sep 19 08:21:08.157021 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.156970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:08.157085 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.157051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:08.157165 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.157142    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:08.157214 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.157186    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:08.157259 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.157222    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret podName:95d065d2-7e01-4e45-83a6-22e2646d868f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:40.157202545 +0000 UTC m=+67.838785969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret") pod "global-pull-secret-syncer-gqvcc" (UID: "95d065d2-7e01-4e45-83a6-22e2646d868f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:08.157259 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.157242    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs podName:5cbb75e1-d58d-41d4-ba14-9e3194e793f8 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:40.157232665 +0000 UTC m=+67.838816087 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs") pod "network-metrics-daemon-kwpz5" (UID: "5cbb75e1-d58d-41d4-ba14-9e3194e793f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:08.262253 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.258657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:08.262253 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.258815    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:21:08.262253 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.258847    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:21:08.262253 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.258862    2577 projected.go:196] Error preparing data for projected volume kube-api-access-b496h for pod openshift-network-diagnostics/network-check-target-p74dm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:08.262253 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.258926    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h podName:ba99474c-b9e8-42c3-8e5d-e899124b24a3 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:40.258907572 +0000 UTC m=+67.940490984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b496h" (UniqueName: "kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h") pod "network-check-target-p74dm" (UID: "ba99474c-b9e8-42c3-8e5d-e899124b24a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:08.562092 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.562052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:08.562699 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.562228    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:08.562699 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:08.562314    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls podName:28834fd2-afab-47de-a085-52668ee84b6f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:09.562290566 +0000 UTC m=+37.243874016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls") pod "dns-default-kdt58" (UID: "28834fd2-afab-47de-a085-52668ee84b6f") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:08.945218 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:08.945188    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5ld78"]
Sep 19 08:21:08.949493 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:08.949461    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe02c1ed_a740_42d1_be31_3290295ef972.slice/crio-28e912973f7fd88df6c258344095eec8def872308d0009dee79744e4da98f7e5 WatchSource:0}: Error finding container 28e912973f7fd88df6c258344095eec8def872308d0009dee79744e4da98f7e5: Status 404 returned error can't find the container with id 28e912973f7fd88df6c258344095eec8def872308d0009dee79744e4da98f7e5
Sep 19 08:21:09.498095 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.497929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:09.498272 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.497930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:09.498272 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.497930    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:09.502201 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.502160    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:09.502201 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.502176    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:09.502201 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.502159    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jjfx\""
Sep 19 08:21:09.502201 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.502194    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:09.502201 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.502198    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vlr4t\""
Sep 19 08:21:09.502491 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.502163    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:09.569997 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.569961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:09.570748 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:09.570144    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:09.570748 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:09.570219    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls podName:28834fd2-afab-47de-a085-52668ee84b6f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:11.570203017 +0000 UTC m=+39.251786428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls") pod "dns-default-kdt58" (UID: "28834fd2-afab-47de-a085-52668ee84b6f") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:09.674347 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.674250    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0649707-f033-46a5-b1ea-595b2a403f01" containerID="4b07bdcb2b77c6f3c9e4b4db7f1363a10d6ce7fc81c496636f8f3e88330c1483" exitCode=0
Sep 19 08:21:09.674347 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.674293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fh57" event={"ID":"a0649707-f033-46a5-b1ea-595b2a403f01","Type":"ContainerDied","Data":"4b07bdcb2b77c6f3c9e4b4db7f1363a10d6ce7fc81c496636f8f3e88330c1483"}
Sep 19 08:21:09.675855 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:09.675823    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5ld78" event={"ID":"be02c1ed-a740-42d1-be31-3290295ef972","Type":"ContainerStarted","Data":"28e912973f7fd88df6c258344095eec8def872308d0009dee79744e4da98f7e5"}
Sep 19 08:21:10.682796 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:10.682741    2577 generic.go:356] "Generic (PLEG): container finished" podID="a0649707-f033-46a5-b1ea-595b2a403f01" containerID="8452e6d401102c6f3d716f360f156b0f245238378701f504a9f95d466a881865" exitCode=0
Sep 19 08:21:10.683267 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:10.682802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fh57" event={"ID":"a0649707-f033-46a5-b1ea-595b2a403f01","Type":"ContainerDied","Data":"8452e6d401102c6f3d716f360f156b0f245238378701f504a9f95d466a881865"}
Sep 19 08:21:11.585843 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:11.585805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:11.585993 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:11.585963    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:11.586040 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:11.586031    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls podName:28834fd2-afab-47de-a085-52668ee84b6f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:15.586015987 +0000 UTC m=+43.267599398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls") pod "dns-default-kdt58" (UID: "28834fd2-afab-47de-a085-52668ee84b6f") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:11.688588 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:11.688555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6fh57" event={"ID":"a0649707-f033-46a5-b1ea-595b2a403f01","Type":"ContainerStarted","Data":"b7bf3d82509590318ca40ce219d0dd2cf11a5049ab8926360548a3d0e7a468b8"}
Sep 19 08:21:11.689958 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:11.689936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5ld78" event={"ID":"be02c1ed-a740-42d1-be31-3290295ef972","Type":"ContainerStarted","Data":"4f864746bb449e56a48c91ec67241fde01260d9a106475764268dc7f9e4b0779"}
Sep 19 08:21:11.713927 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:11.713884    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6fh57" podStartSLOduration=9.683017197 podStartE2EDuration="37.713868971s" totalImagesPullingTime="28.030851774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.71250612 +0000 UTC m=+39.394089554" watchObservedRunningTime="2026-09-19 08:21:11.713868971 +0000 UTC m=+39.395452403"
Sep 19 08:21:11.735377 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:11.735322    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5ld78" podStartSLOduration=2.279461259 podStartE2EDuration="4.735309712s" totalImagesPullingTime="2.455848453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.734987231 +0000 UTC m=+39.416570665" watchObservedRunningTime="2026-09-19 08:21:11.735309712 +0000 UTC m=+39.416893145"
Sep 19 08:21:15.611590 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:15.611557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:15.611953 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:15.611661    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:15.611953 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:15.611709    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls podName:28834fd2-afab-47de-a085-52668ee84b6f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.611695286 +0000 UTC m=+51.293278697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls") pod "dns-default-kdt58" (UID: "28834fd2-afab-47de-a085-52668ee84b6f") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:16.140400 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.140357    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"]
Sep 19 08:21:16.143066 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.143039    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.143983 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.143964    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"]
Sep 19 08:21:16.146504 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.146479    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 19 08:21:16.146620 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.146526    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 19 08:21:16.146620 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.146570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:16.146735 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.146622    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:21:16.146735 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.146622    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-v94hn\""
Sep 19 08:21:16.146735 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.146570    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 19 08:21:16.149041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.149014    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 19 08:21:16.149138 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.149046    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:21:16.149198 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.149137    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 19 08:21:16.149255 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.149200    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nm968\""
Sep 19 08:21:16.152954 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.152936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"]
Sep 19 08:21:16.156287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.156266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"]
Sep 19 08:21:16.277861 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.277832    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"]
Sep 19 08:21:16.280652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.280635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:16.283335 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.283310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rd6k8\""
Sep 19 08:21:16.283335 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.283325    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 19 08:21:16.283525 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.283406    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 19 08:21:16.292671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.292647    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"]
Sep 19 08:21:16.317402 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.317348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxl2\" (UniqueName: \"kubernetes.io/projected/dde16ec8-2ee3-48a3-a755-b8d40692076b-kube-api-access-fnxl2\") pod \"kube-storage-version-migrator-operator-55889f7b47-48wml\" (UID: \"dde16ec8-2ee3-48a3-a755-b8d40692076b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.317570 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.317438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxl8b\" (UniqueName: \"kubernetes.io/projected/f7d87010-042d-4827-9166-9a5c7e3da944-kube-api-access-mxl8b\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:16.317570 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.317493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dde16ec8-2ee3-48a3-a755-b8d40692076b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-48wml\" (UID: \"dde16ec8-2ee3-48a3-a755-b8d40692076b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.317570 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.317509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dde16ec8-2ee3-48a3-a755-b8d40692076b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-48wml\" (UID: \"dde16ec8-2ee3-48a3-a755-b8d40692076b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.317570 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.317558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:16.369357 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.369326    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"]
Sep 19 08:21:16.372034 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.372013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.373231 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.373204    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"]
Sep 19 08:21:16.374581 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.374566    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:21:16.375156 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.375133    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 19 08:21:16.375306 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.375141    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 19 08:21:16.375306 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.375303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8wzfz\""
Sep 19 08:21:16.375817 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.375802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:16.384581 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.384564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 19 08:21:16.384994 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.384978    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 08:21:16.385127 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.385109    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 19 08:21:16.385690 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.385674    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57d7bbbb77-r92gn"]
Sep 19 08:21:16.386090 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.386071    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 08:21:16.387158 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.387139    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wkjxz\""
Sep 19 08:21:16.387304 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.387275    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 19 08:21:16.388616 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.388591    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f9d67fdf8-x2dbm"]
Sep 19 08:21:16.388791 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.388702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.392490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.392441    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 08:21:16.392643 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.392443    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 08:21:16.392643 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.392534    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 08:21:16.392643 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.392599    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hr2s5\""
Sep 19 08:21:16.393511 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.393490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"]
Sep 19 08:21:16.393772 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.393604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.395915 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.395897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 19 08:21:16.396166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.396149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 19 08:21:16.396399 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.396384    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 19 08:21:16.396460 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.396400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lwjc6\""
Sep 19 08:21:16.396529 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.396516    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 19 08:21:16.396582 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.396518    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 19 08:21:16.397451 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.397432    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 19 08:21:16.398843 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.398824    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 08:21:16.406860 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.406826    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57d7bbbb77-r92gn"]
Sep 19 08:21:16.407573 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.407556    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f9d67fdf8-x2dbm"]
Sep 19 08:21:16.408738 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.408720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"]
Sep 19 08:21:16.418123 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.418105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dde16ec8-2ee3-48a3-a755-b8d40692076b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-48wml\" (UID: \"dde16ec8-2ee3-48a3-a755-b8d40692076b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.418215 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.418131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dde16ec8-2ee3-48a3-a755-b8d40692076b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-48wml\" (UID: \"dde16ec8-2ee3-48a3-a755-b8d40692076b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.418215 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.418153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:16.418215 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.418174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxl2\" (UniqueName: \"kubernetes.io/projected/dde16ec8-2ee3-48a3-a755-b8d40692076b-kube-api-access-fnxl2\") pod \"kube-storage-version-migrator-operator-55889f7b47-48wml\" (UID: \"dde16ec8-2ee3-48a3-a755-b8d40692076b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.418332 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.418206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e873fa1c-aba8-41f9-9f87-66b3b4439306-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:16.418332 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.418262    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:16.418332 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.418272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:16.418332 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.418317    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls podName:f7d87010-042d-4827-9166-9a5c7e3da944 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:16.918301966 +0000 UTC m=+44.599885390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-hlxkx" (UID: "f7d87010-042d-4827-9166-9a5c7e3da944") : secret "samples-operator-tls" not found
Sep 19 08:21:16.418501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.418431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxl8b\" (UniqueName: \"kubernetes.io/projected/f7d87010-042d-4827-9166-9a5c7e3da944-kube-api-access-mxl8b\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:16.418754 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.418738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dde16ec8-2ee3-48a3-a755-b8d40692076b-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-48wml\" (UID: \"dde16ec8-2ee3-48a3-a755-b8d40692076b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.421725 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.421705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dde16ec8-2ee3-48a3-a755-b8d40692076b-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-48wml\" (UID: \"dde16ec8-2ee3-48a3-a755-b8d40692076b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.433135 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.433114    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxl8b\" (UniqueName: \"kubernetes.io/projected/f7d87010-042d-4827-9166-9a5c7e3da944-kube-api-access-mxl8b\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:16.437765 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.437745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxl2\" (UniqueName: \"kubernetes.io/projected/dde16ec8-2ee3-48a3-a755-b8d40692076b-kube-api-access-fnxl2\") pod \"kube-storage-version-migrator-operator-55889f7b47-48wml\" (UID: \"dde16ec8-2ee3-48a3-a755-b8d40692076b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.454244 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.454225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"
Sep 19 08:21:16.519631 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-certificates\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.519747 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-trusted-ca\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.519747 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-installation-pull-secrets\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.519747 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2b8j\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-kube-api-access-h2b8j\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.519747 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.519747 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:16.519903 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a7f5c37-2015-4424-b262-b8a1729653e2-config\") pod \"service-ca-operator-57cdc4b768-6q6kq\" (UID: \"1a7f5c37-2015-4424-b262-b8a1729653e2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.519903 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-default-certificate\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.519903 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.519903 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e873fa1c-aba8-41f9-9f87-66b3b4439306-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:16.519903 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-stats-auth\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.520041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf95z\" (UniqueName: \"kubernetes.io/projected/1a7f5c37-2015-4424-b262-b8a1729653e2-kube-api-access-zf95z\") pod \"service-ca-operator-57cdc4b768-6q6kq\" (UID: \"1a7f5c37-2015-4424-b262-b8a1729653e2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.520041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzh9\" (UniqueName: \"kubernetes.io/projected/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-kube-api-access-fqzh9\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:16.520041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.519986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-image-registry-private-configuration\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.520132 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.520054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njm79\" (UniqueName: \"kubernetes.io/projected/ddfd0060-8f85-422c-85c1-9570edc61e0d-kube-api-access-njm79\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.520132 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.520081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0e6b508-19eb-4062-afd9-ad5260f3f153-ca-trust-extracted\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.520132 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.520124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:16.520218 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.520147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.520218 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.520170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-bound-sa-token\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.520287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.520221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a7f5c37-2015-4424-b262-b8a1729653e2-serving-cert\") pod \"service-ca-operator-57cdc4b768-6q6kq\" (UID: \"1a7f5c37-2015-4424-b262-b8a1729653e2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.520287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.520253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:16.520718 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.520391    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:16.520718 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.520457    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert podName:e873fa1c-aba8-41f9-9f87-66b3b4439306 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:17.020439491 +0000 UTC m=+44.702022910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ptbj8" (UID: "e873fa1c-aba8-41f9-9f87-66b3b4439306") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:16.520829 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.520764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e873fa1c-aba8-41f9-9f87-66b3b4439306-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:16.621244 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njm79\" (UniqueName: \"kubernetes.io/projected/ddfd0060-8f85-422c-85c1-9570edc61e0d-kube-api-access-njm79\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.621244 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0e6b508-19eb-4062-afd9-ad5260f3f153-ca-trust-extracted\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.621436    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-bound-sa-token\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.621501    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls podName:e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:17.121479858 +0000 UTC m=+44.803063290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xw5l6" (UID: "e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a7f5c37-2015-4424-b262-b8a1729653e2-serving-cert\") pod \"service-ca-operator-57cdc4b768-6q6kq\" (UID: \"1a7f5c37-2015-4424-b262-b8a1729653e2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.621589    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:17.121569946 +0000 UTC m=+44.803153361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-certificates\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-trusted-ca\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-installation-pull-secrets\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.621818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0e6b508-19eb-4062-afd9-ad5260f3f153-ca-trust-extracted\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2b8j\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-kube-api-access-h2b8j\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.621993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a7f5c37-2015-4424-b262-b8a1729653e2-config\") pod \"service-ca-operator-57cdc4b768-6q6kq\" (UID: \"1a7f5c37-2015-4424-b262-b8a1729653e2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.622025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-default-certificate\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.622047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.622101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-stats-auth\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.622124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf95z\" (UniqueName: \"kubernetes.io/projected/1a7f5c37-2015-4424-b262-b8a1729653e2-kube-api-access-zf95z\") pod \"service-ca-operator-57cdc4b768-6q6kq\" (UID: \"1a7f5c37-2015-4424-b262-b8a1729653e2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.622152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzh9\" (UniqueName: \"kubernetes.io/projected/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-kube-api-access-fqzh9\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.622175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-image-registry-private-configuration\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.622201    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.622217    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7bbbb77-r92gn: secret "image-registry-tls" not found
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.622263    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls podName:d0e6b508-19eb-4062-afd9-ad5260f3f153 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:17.12224695 +0000 UTC m=+44.803830386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls") pod "image-registry-57d7bbbb77-r92gn" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153") : secret "image-registry-tls" not found
Sep 19 08:21:16.622393 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.622400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-certificates\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.623097 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.622808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:16.623097 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.622819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a7f5c37-2015-4424-b262-b8a1729653e2-config\") pod \"service-ca-operator-57cdc4b768-6q6kq\" (UID: \"1a7f5c37-2015-4424-b262-b8a1729653e2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.623209 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.623101    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-trusted-ca\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.623209 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.623131    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:16.623209 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.623190    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:17.123174354 +0000 UTC m=+44.804757769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : secret "router-metrics-certs-default" not found
Sep 19 08:21:16.624149 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.624121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a7f5c37-2015-4424-b262-b8a1729653e2-serving-cert\") pod \"service-ca-operator-57cdc4b768-6q6kq\" (UID: \"1a7f5c37-2015-4424-b262-b8a1729653e2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.624461 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.624435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-installation-pull-secrets\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.624764 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.624740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-image-registry-private-configuration\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.625111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.625089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-default-certificate\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.625227 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.625211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-stats-auth\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.630173 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.630148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njm79\" (UniqueName: \"kubernetes.io/projected/ddfd0060-8f85-422c-85c1-9570edc61e0d-kube-api-access-njm79\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:16.631180 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.631153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzh9\" (UniqueName: \"kubernetes.io/projected/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-kube-api-access-fqzh9\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:16.631730 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.631704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2b8j\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-kube-api-access-h2b8j\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.631827 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.631735    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf95z\" (UniqueName: \"kubernetes.io/projected/1a7f5c37-2015-4424-b262-b8a1729653e2-kube-api-access-zf95z\") pod \"service-ca-operator-57cdc4b768-6q6kq\" (UID: \"1a7f5c37-2015-4424-b262-b8a1729653e2\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.631827 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.631782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-bound-sa-token\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:16.682253 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.682181    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"
Sep 19 08:21:16.725692 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.725660    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml"]
Sep 19 08:21:16.729055 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:16.729024    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde16ec8_2ee3_48a3_a755_b8d40692076b.slice/crio-0f5cfe02fb4db9a59fce20b9da6029e96759e8bf5d7e8fd609f31cd720c8fc0d WatchSource:0}: Error finding container 0f5cfe02fb4db9a59fce20b9da6029e96759e8bf5d7e8fd609f31cd720c8fc0d: Status 404 returned error can't find the container with id 0f5cfe02fb4db9a59fce20b9da6029e96759e8bf5d7e8fd609f31cd720c8fc0d
Sep 19 08:21:16.925912 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.925875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:16.926077 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.925988    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:16.926077 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:16.926041    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls podName:f7d87010-042d-4827-9166-9a5c7e3da944 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:17.926027691 +0000 UTC m=+45.607611102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-hlxkx" (UID: "f7d87010-042d-4827-9166-9a5c7e3da944") : secret "samples-operator-tls" not found
Sep 19 08:21:16.957036 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:16.956956    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq"]
Sep 19 08:21:16.960512 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:16.960487    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7f5c37_2015_4424_b262_b8a1729653e2.slice/crio-e60125488ca600332141e46a50a1ae257b63f6efa4b75c86062102a30a0c73ec WatchSource:0}: Error finding container e60125488ca600332141e46a50a1ae257b63f6efa4b75c86062102a30a0c73ec: Status 404 returned error can't find the container with id e60125488ca600332141e46a50a1ae257b63f6efa4b75c86062102a30a0c73ec
Sep 19 08:21:17.027008 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:17.026979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:17.027162 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.027142    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:17.027223 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.027214    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert podName:e873fa1c-aba8-41f9-9f87-66b3b4439306 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:18.027197859 +0000 UTC m=+45.708781271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ptbj8" (UID: "e873fa1c-aba8-41f9-9f87-66b3b4439306") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:17.128383 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:17.128327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:17.128524 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:17.128391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:17.128524 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:17.128465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:17.128524 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.128492    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:17.128524 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:17.128502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:17.128666 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.128557    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls podName:e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:18.128537807 +0000 UTC m=+45.810121238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xw5l6" (UID: "e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:17.128666 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.128575    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:18.128565991 +0000 UTC m=+45.810149409 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:17.128666 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.128590    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:17.128666 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.128621    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:18.128611507 +0000 UTC m=+45.810194920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : secret "router-metrics-certs-default" not found
Sep 19 08:21:17.128666 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.128634    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:17.128666 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.128655    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7bbbb77-r92gn: secret "image-registry-tls" not found
Sep 19 08:21:17.128847 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.128690    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls podName:d0e6b508-19eb-4062-afd9-ad5260f3f153 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:18.12867756 +0000 UTC m=+45.810260970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls") pod "image-registry-57d7bbbb77-r92gn" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153") : secret "image-registry-tls" not found
Sep 19 08:21:17.706040 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:17.706004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq" event={"ID":"1a7f5c37-2015-4424-b262-b8a1729653e2","Type":"ContainerStarted","Data":"e60125488ca600332141e46a50a1ae257b63f6efa4b75c86062102a30a0c73ec"}
Sep 19 08:21:17.707627 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:17.707601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml" event={"ID":"dde16ec8-2ee3-48a3-a755-b8d40692076b","Type":"ContainerStarted","Data":"0f5cfe02fb4db9a59fce20b9da6029e96759e8bf5d7e8fd609f31cd720c8fc0d"}
Sep 19 08:21:17.936371 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:17.936327    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:17.936576 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.936529    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:17.936645 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:17.936609    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls podName:f7d87010-042d-4827-9166-9a5c7e3da944 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:19.93658768 +0000 UTC m=+47.618171111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-hlxkx" (UID: "f7d87010-042d-4827-9166-9a5c7e3da944") : secret "samples-operator-tls" not found
Sep 19 08:21:18.037642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:18.037554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:18.037787 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.037727    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:18.037827 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.037793    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert podName:e873fa1c-aba8-41f9-9f87-66b3b4439306 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:20.037777584 +0000 UTC m=+47.719361001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ptbj8" (UID: "e873fa1c-aba8-41f9-9f87-66b3b4439306") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:18.138494 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:18.138450    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:18.138669 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:18.138512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:18.138669 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:18.138602    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:18.138669 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:18.138631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:18.138669 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.138645    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:18.138669 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.138671    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7bbbb77-r92gn: secret "image-registry-tls" not found
Sep 19 08:21:18.138915 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.138699    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:18.138915 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.138732    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls podName:d0e6b508-19eb-4062-afd9-ad5260f3f153 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:20.138712739 +0000 UTC m=+47.820296167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls") pod "image-registry-57d7bbbb77-r92gn" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153") : secret "image-registry-tls" not found
Sep 19 08:21:18.138915 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.138769    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:18.138915 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.138798    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:20.138781571 +0000 UTC m=+47.820364987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:18.138915 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.138819    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:20.138808614 +0000 UTC m=+47.820392024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : secret "router-metrics-certs-default" not found
Sep 19 08:21:18.138915 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:18.138842    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls podName:e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:20.13883377 +0000 UTC m=+47.820417185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xw5l6" (UID: "e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:19.713748 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:19.713716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq" event={"ID":"1a7f5c37-2015-4424-b262-b8a1729653e2","Type":"ContainerStarted","Data":"4cad9a304ed82311bc324aea2acc093acd254c7a6bd4a8f1113b53c2528d868d"}
Sep 19 08:21:19.715297 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:19.715274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml" event={"ID":"dde16ec8-2ee3-48a3-a755-b8d40692076b","Type":"ContainerStarted","Data":"41c60a10396da3f5f1038c8f311b3c9b2e8c9e56c3f920643dbc97e2f1a73335"}
Sep 19 08:21:19.730230 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:19.730182    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq" podStartSLOduration=1.5317192899999998 podStartE2EDuration="3.730169412s" totalImagesPullingTime="2.198450122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:19.729455726 +0000 UTC m=+47.411039164" watchObservedRunningTime="2026-09-19 08:21:19.730169412 +0000 UTC m=+47.411752842"
Sep 19 08:21:19.747666 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:19.747623    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml" podStartSLOduration=1.320556661 podStartE2EDuration="3.747609902s" totalImagesPullingTime="2.427053241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:19.747123521 +0000 UTC m=+47.428706954" watchObservedRunningTime="2026-09-19 08:21:19.747609902 +0000 UTC m=+47.429193335"
Sep 19 08:21:19.955599 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:19.955559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:19.955751 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:19.955724    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:19.955809 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:19.955793    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls podName:f7d87010-042d-4827-9166-9a5c7e3da944 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.955777783 +0000 UTC m=+51.637361194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-hlxkx" (UID: "f7d87010-042d-4827-9166-9a5c7e3da944") : secret "samples-operator-tls" not found
Sep 19 08:21:20.056338 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:20.056247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:20.056494 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.056435    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:20.056529 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.056520    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert podName:e873fa1c-aba8-41f9-9f87-66b3b4439306 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.056497599 +0000 UTC m=+51.738081016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ptbj8" (UID: "e873fa1c-aba8-41f9-9f87-66b3b4439306") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:20.156778 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:20.156745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:20.156922 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:20.156782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:20.156922 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:20.156829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:20.156922 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:20.156852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:20.157022 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.156922    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:20.157022 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.156955    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:20.157022 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.156971    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7bbbb77-r92gn: secret "image-registry-tls" not found
Sep 19 08:21:20.157022 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.156927    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:20.157022 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.157001    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls podName:e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.156977996 +0000 UTC m=+51.838561422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xw5l6" (UID: "e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:20.157022 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.157020    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.157009566 +0000 UTC m=+51.838592983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:20.157246 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.157034    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls podName:d0e6b508-19eb-4062-afd9-ad5260f3f153 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.157026185 +0000 UTC m=+51.838609602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls") pod "image-registry-57d7bbbb77-r92gn" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153") : secret "image-registry-tls" not found
Sep 19 08:21:20.157246 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:20.157118    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.157109645 +0000 UTC m=+51.838693059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : secret "router-metrics-certs-default" not found
Sep 19 08:21:22.863156 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.863123    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9ldsl"]
Sep 19 08:21:22.877474 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.877452    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9ldsl"]
Sep 19 08:21:22.877580 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.877546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:22.880157 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.880136    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 08:21:22.880157 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.880134    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 08:21:22.880351 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.880276    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 08:21:22.880351 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.880291    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 08:21:22.880351 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.880293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qqzk5\""
Sep 19 08:21:22.981753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.981720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2272806d-5340-4bd6-b669-8f5f2a74fb64-crio-socket\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:22.981911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.981777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2272806d-5340-4bd6-b669-8f5f2a74fb64-data-volume\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:22.981911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.981844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2x5j\" (UniqueName: \"kubernetes.io/projected/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-api-access-g2x5j\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:22.981911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.981878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:22.982007 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:22.981925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.082877 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.082841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2x5j\" (UniqueName: \"kubernetes.io/projected/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-api-access-g2x5j\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.083022 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.082882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.083022 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.082955    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:23.083022 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.083010    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-rbac-proxy-cm podName:2272806d-5340-4bd6-b669-8f5f2a74fb64 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.582990433 +0000 UTC m=+51.264573846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9ldsl" (UID: "2272806d-5340-4bd6-b669-8f5f2a74fb64") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:23.083151 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.083042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.083151 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.083112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2272806d-5340-4bd6-b669-8f5f2a74fb64-crio-socket\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.083231 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.083167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2272806d-5340-4bd6-b669-8f5f2a74fb64-data-volume\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.083231 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.083208    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:23.083297 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.083252    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls podName:2272806d-5340-4bd6-b669-8f5f2a74fb64 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.583240579 +0000 UTC m=+51.264824021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9ldsl" (UID: "2272806d-5340-4bd6-b669-8f5f2a74fb64") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:23.083349 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.083329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2272806d-5340-4bd6-b669-8f5f2a74fb64-crio-socket\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.083493 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.083479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2272806d-5340-4bd6-b669-8f5f2a74fb64-data-volume\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.095827 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.095804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2x5j\" (UniqueName: \"kubernetes.io/projected/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-api-access-g2x5j\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.588422 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.588380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.588662 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.588445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:23.588662 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.588505    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:23.588662 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.588579    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-rbac-proxy-cm podName:2272806d-5340-4bd6-b669-8f5f2a74fb64 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.588563244 +0000 UTC m=+52.270146660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9ldsl" (UID: "2272806d-5340-4bd6-b669-8f5f2a74fb64") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:23.588662 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.588606    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:23.588808 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.588672    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls podName:2272806d-5340-4bd6-b669-8f5f2a74fb64 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.588657099 +0000 UTC m=+52.270240512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9ldsl" (UID: "2272806d-5340-4bd6-b669-8f5f2a74fb64") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:23.688888 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.688851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:23.689055 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.689004    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:23.689107 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.689068    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls podName:28834fd2-afab-47de-a085-52668ee84b6f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.689052704 +0000 UTC m=+67.370636115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls") pod "dns-default-kdt58" (UID: "28834fd2-afab-47de-a085-52668ee84b6f") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:23.991623 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:23.991579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:23.992026 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.991730    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 19 08:21:23.992026 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:23.991797    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls podName:f7d87010-042d-4827-9166-9a5c7e3da944 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:31.991780587 +0000 UTC m=+59.673363998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-hlxkx" (UID: "f7d87010-042d-4827-9166-9a5c7e3da944") : secret "samples-operator-tls" not found
Sep 19 08:21:24.092405 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:24.092355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:24.092564 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.092509    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:24.092603 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.092576    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert podName:e873fa1c-aba8-41f9-9f87-66b3b4439306 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:32.092560888 +0000 UTC m=+59.774144298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ptbj8" (UID: "e873fa1c-aba8-41f9-9f87-66b3b4439306") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:24.193125 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:24.193091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:24.193267 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:24.193131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:24.193267 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:24.193181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:24.193267 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:24.193205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:24.193267 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.193257    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:24.193494 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.193307    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:24.193494 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.193317    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7bbbb77-r92gn: secret "image-registry-tls" not found
Sep 19 08:21:24.193494 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.193332    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls podName:e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:32.193311374 +0000 UTC m=+59.874894805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xw5l6" (UID: "e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:24.193494 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.193351    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:32.193342175 +0000 UTC m=+59.874925599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:24.193494 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.193397    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 19 08:21:24.193494 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.193427    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls podName:d0e6b508-19eb-4062-afd9-ad5260f3f153 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:32.193417396 +0000 UTC m=+59.875000810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls") pod "image-registry-57d7bbbb77-r92gn" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153") : secret "image-registry-tls" not found
Sep 19 08:21:24.193494 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.193444    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:32.193436004 +0000 UTC m=+59.875019417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : secret "router-metrics-certs-default" not found
Sep 19 08:21:24.597468 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:24.597438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:24.597653 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:24.597498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:24.597653 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.597584    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:24.597747 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:24.597725    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls podName:2272806d-5340-4bd6-b669-8f5f2a74fb64 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:26.597698969 +0000 UTC m=+54.279282390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9ldsl" (UID: "2272806d-5340-4bd6-b669-8f5f2a74fb64") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:24.597991 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:24.597976    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2272806d-5340-4bd6-b669-8f5f2a74fb64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:25.220454 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:25.220423    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ltmhg_a791e266-0859-46a0-920f-dcec31e7a326/dns-node-resolver/0.log"
Sep 19 08:21:26.019459 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:26.019429    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzm5f_c1c234fa-2c7e-43db-9f11-be533024f6a7/node-ca/0.log"
Sep 19 08:21:26.615480 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:26.615441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:26.615897 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:26.615574    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:26.615897 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:26.615630    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls podName:2272806d-5340-4bd6-b669-8f5f2a74fb64 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:30.615615115 +0000 UTC m=+58.297198540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9ldsl" (UID: "2272806d-5340-4bd6-b669-8f5f2a74fb64") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:26.818703 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:26.818672    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5ld78_be02c1ed-a740-42d1-be31-3290295ef972/serve-healthcheck-canary/0.log"
Sep 19 08:21:27.619160 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:27.619123    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-48wml_dde16ec8-2ee3-48a3-a755-b8d40692076b/kube-storage-version-migrator-operator/0.log"
Sep 19 08:21:30.651385 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:30.651332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:30.651847 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:30.651486    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:30.651847 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:30.651556    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls podName:2272806d-5340-4bd6-b669-8f5f2a74fb64 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:38.651539031 +0000 UTC m=+66.333122442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9ldsl" (UID: "2272806d-5340-4bd6-b669-8f5f2a74fb64") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:32.065210 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.065158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:32.067717 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.067688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7d87010-042d-4827-9166-9a5c7e3da944-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-hlxkx\" (UID: \"f7d87010-042d-4827-9166-9a5c7e3da944\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:32.166244 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.166201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:32.166438 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:32.166346    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 19 08:21:32.166485 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:32.166441    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert podName:e873fa1c-aba8-41f9-9f87-66b3b4439306 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:48.1664238 +0000 UTC m=+75.848007228 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-ptbj8" (UID: "e873fa1c-aba8-41f9-9f87-66b3b4439306") : secret "networking-console-plugin-cert" not found
Sep 19 08:21:32.267017 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.266979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:32.267190 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.267030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:32.267190 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.267107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:32.267190 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.267133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:32.267311 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:32.267248    2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:32.267311 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:32.267265    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle podName:ddfd0060-8f85-422c-85c1-9570edc61e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:48.267246066 +0000 UTC m=+75.948829495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle") pod "router-default-5f9d67fdf8-x2dbm" (UID: "ddfd0060-8f85-422c-85c1-9570edc61e0d") : configmap references non-existent config key: service-ca.crt
Sep 19 08:21:32.267432 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:21:32.267334    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls podName:e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:48.267314961 +0000 UTC m=+75.948898373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-xw5l6" (UID: "e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36") : secret "cluster-monitoring-operator-tls" not found
Sep 19 08:21:32.269428 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.269394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddfd0060-8f85-422c-85c1-9570edc61e0d-metrics-certs\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:32.269428 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.269394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") pod \"image-registry-57d7bbbb77-r92gn\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") " pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:32.301780 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.301741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:32.360054 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.360018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"
Sep 19 08:21:32.603938 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.603907    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57d7bbbb77-r92gn"]
Sep 19 08:21:32.607051 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:32.607018    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e6b508_19eb_4062_afd9_ad5260f3f153.slice/crio-6759e325b7da221aa47c460fe9245b8aca11d2ea93668347f1f614c946115ca4 WatchSource:0}: Error finding container 6759e325b7da221aa47c460fe9245b8aca11d2ea93668347f1f614c946115ca4: Status 404 returned error can't find the container with id 6759e325b7da221aa47c460fe9245b8aca11d2ea93668347f1f614c946115ca4
Sep 19 08:21:32.650701 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.650661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx"]
Sep 19 08:21:32.747635 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.747594    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx" event={"ID":"f7d87010-042d-4827-9166-9a5c7e3da944","Type":"ContainerStarted","Data":"7de7d68dba86da518796f971672f53299667fd9f0d3070bd1ecd9a9e602d75cd"}
Sep 19 08:21:32.749183 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.749157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn" event={"ID":"d0e6b508-19eb-4062-afd9-ad5260f3f153","Type":"ContainerStarted","Data":"809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028"}
Sep 19 08:21:32.749322 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.749196    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn" event={"ID":"d0e6b508-19eb-4062-afd9-ad5260f3f153","Type":"ContainerStarted","Data":"6759e325b7da221aa47c460fe9245b8aca11d2ea93668347f1f614c946115ca4"}
Sep 19 08:21:32.749322 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.749224    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:21:32.767714 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:32.767656    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn" podStartSLOduration=16.767638535 podStartE2EDuration="16.767638535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:32.767348904 +0000 UTC m=+60.448932361" watchObservedRunningTime="2026-09-19 08:21:32.767638535 +0000 UTC m=+60.449221970"
Sep 19 08:21:34.669438 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:34.669411    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6np9d"
Sep 19 08:21:34.757531 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:34.757489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx" event={"ID":"f7d87010-042d-4827-9166-9a5c7e3da944","Type":"ContainerStarted","Data":"d923c36cf09a820519adc4b745288f9fd1c873bd160446bfa22c20802da2a8a5"}
Sep 19 08:21:34.757726 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:34.757590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx" event={"ID":"f7d87010-042d-4827-9166-9a5c7e3da944","Type":"ContainerStarted","Data":"0ad108404e827bed6f334db6d5d3bdc5b24ca7ddbaf92412342f27d0e75403a1"}
Sep 19 08:21:34.774586 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:34.774524    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-hlxkx" podStartSLOduration=16.866086455 podStartE2EDuration="18.774503761s" totalImagesPullingTime="1.908417306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:34.773510738 +0000 UTC m=+62.455094174" watchObservedRunningTime="2026-09-19 08:21:34.774503761 +0000 UTC m=+62.456087194"
Sep 19 08:21:38.727418 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:38.727355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:38.729709 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:38.729684    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2272806d-5340-4bd6-b669-8f5f2a74fb64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9ldsl\" (UID: \"2272806d-5340-4bd6-b669-8f5f2a74fb64\") " pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:38.789471 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:38.789442    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qqzk5\""
Sep 19 08:21:38.797640 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:38.797616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9ldsl"
Sep 19 08:21:39.073510 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:39.073432    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9ldsl"]
Sep 19 08:21:39.076143 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:39.076117    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2272806d_5340_4bd6_b669_8f5f2a74fb64.slice/crio-430d7a6db3153adab1c1cea4928f11f60c1bb47209a96d180f34806e6e3d6393 WatchSource:0}: Error finding container 430d7a6db3153adab1c1cea4928f11f60c1bb47209a96d180f34806e6e3d6393: Status 404 returned error can't find the container with id 430d7a6db3153adab1c1cea4928f11f60c1bb47209a96d180f34806e6e3d6393
Sep 19 08:21:39.737569 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:39.737521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:39.740177 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:39.740148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/28834fd2-afab-47de-a085-52668ee84b6f-metrics-tls\") pod \"dns-default-kdt58\" (UID: \"28834fd2-afab-47de-a085-52668ee84b6f\") " pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:39.775117 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:39.775027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9ldsl" event={"ID":"2272806d-5340-4bd6-b669-8f5f2a74fb64","Type":"ContainerStarted","Data":"d503135c7a3867d5793c0c5b9a1bd7e2fd135aeab2e1a449e5a4bd9a6fb3f358"}
Sep 19 08:21:39.775117 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:39.775089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9ldsl" event={"ID":"2272806d-5340-4bd6-b669-8f5f2a74fb64","Type":"ContainerStarted","Data":"430d7a6db3153adab1c1cea4928f11f60c1bb47209a96d180f34806e6e3d6393"}
Sep 19 08:21:39.953249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:39.953225    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wm7fh\""
Sep 19 08:21:39.961485 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:39.961461    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:40.231250 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.231219    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kdt58"]
Sep 19 08:21:40.234488 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:40.234458    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28834fd2_afab_47de_a085_52668ee84b6f.slice/crio-49e5a46209a99a4ed676794c909ad2e09bb5514764e4718260b4338b4dea78d3 WatchSource:0}: Error finding container 49e5a46209a99a4ed676794c909ad2e09bb5514764e4718260b4338b4dea78d3: Status 404 returned error can't find the container with id 49e5a46209a99a4ed676794c909ad2e09bb5514764e4718260b4338b4dea78d3
Sep 19 08:21:40.240861 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.240841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:40.240945 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.240876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:40.243508 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.243485    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:40.243601 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.243510    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:40.253774 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.253753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cbb75e1-d58d-41d4-ba14-9e3194e793f8-metrics-certs\") pod \"network-metrics-daemon-kwpz5\" (UID: \"5cbb75e1-d58d-41d4-ba14-9e3194e793f8\") " pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:40.253774 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.253770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/95d065d2-7e01-4e45-83a6-22e2646d868f-original-pull-secret\") pod \"global-pull-secret-syncer-gqvcc\" (UID: \"95d065d2-7e01-4e45-83a6-22e2646d868f\") " pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:40.341648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.341539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:40.344727 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.344703    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:40.354213 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.354189    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:40.364600 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.364576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b496h\" (UniqueName: \"kubernetes.io/projected/ba99474c-b9e8-42c3-8e5d-e899124b24a3-kube-api-access-b496h\") pod \"network-check-target-p74dm\" (UID: \"ba99474c-b9e8-42c3-8e5d-e899124b24a3\") " pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:40.409338 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.409304    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vlr4t\""
Sep 19 08:21:40.415190 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.415171    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jjfx\""
Sep 19 08:21:40.417252 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.417234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kwpz5"
Sep 19 08:21:40.417316 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.417266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqvcc"
Sep 19 08:21:40.423103 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.423073    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:40.704211 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.703978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqvcc"]
Sep 19 08:21:40.705279 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.705249    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p74dm"]
Sep 19 08:21:40.706171 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.706149    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kwpz5"]
Sep 19 08:21:40.706587 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:40.706558    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba99474c_b9e8_42c3_8e5d_e899124b24a3.slice/crio-79f9a2351d1ed4b4e521be7501f9f50224eb3a7a0176b687140b3a722201aa78 WatchSource:0}: Error finding container 79f9a2351d1ed4b4e521be7501f9f50224eb3a7a0176b687140b3a722201aa78: Status 404 returned error can't find the container with id 79f9a2351d1ed4b4e521be7501f9f50224eb3a7a0176b687140b3a722201aa78
Sep 19 08:21:40.707412 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:40.707382    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d065d2_7e01_4e45_83a6_22e2646d868f.slice/crio-b450352aefe1ab5271794e4c0e686d8dd2fe45c95942419782edcfaeff383af3 WatchSource:0}: Error finding container b450352aefe1ab5271794e4c0e686d8dd2fe45c95942419782edcfaeff383af3: Status 404 returned error can't find the container with id b450352aefe1ab5271794e4c0e686d8dd2fe45c95942419782edcfaeff383af3
Sep 19 08:21:40.708199 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:40.708169    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbb75e1_d58d_41d4_ba14_9e3194e793f8.slice/crio-9a1b79088170ec518065003b79caa4b894af629755cb162acb5aa495d92624b4 WatchSource:0}: Error finding container 9a1b79088170ec518065003b79caa4b894af629755cb162acb5aa495d92624b4: Status 404 returned error can't find the container with id 9a1b79088170ec518065003b79caa4b894af629755cb162acb5aa495d92624b4
Sep 19 08:21:40.781805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.781766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gqvcc" event={"ID":"95d065d2-7e01-4e45-83a6-22e2646d868f","Type":"ContainerStarted","Data":"b450352aefe1ab5271794e4c0e686d8dd2fe45c95942419782edcfaeff383af3"}
Sep 19 08:21:40.783304 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.783271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kwpz5" event={"ID":"5cbb75e1-d58d-41d4-ba14-9e3194e793f8","Type":"ContainerStarted","Data":"9a1b79088170ec518065003b79caa4b894af629755cb162acb5aa495d92624b4"}
Sep 19 08:21:40.785561 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.785529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9ldsl" event={"ID":"2272806d-5340-4bd6-b669-8f5f2a74fb64","Type":"ContainerStarted","Data":"28fda8eae3888a613c509c581b710db8a724633c2bb46a64427cc4ea0963261a"}
Sep 19 08:21:40.786866 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.786840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p74dm" event={"ID":"ba99474c-b9e8-42c3-8e5d-e899124b24a3","Type":"ContainerStarted","Data":"79f9a2351d1ed4b4e521be7501f9f50224eb3a7a0176b687140b3a722201aa78"}
Sep 19 08:21:40.788222 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:40.788189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kdt58" event={"ID":"28834fd2-afab-47de-a085-52668ee84b6f","Type":"ContainerStarted","Data":"49e5a46209a99a4ed676794c909ad2e09bb5514764e4718260b4338b4dea78d3"}
Sep 19 08:21:41.797618 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:41.797531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9ldsl" event={"ID":"2272806d-5340-4bd6-b669-8f5f2a74fb64","Type":"ContainerStarted","Data":"91be134f14d584e887f57a822780aef0aac4816178f85c7619e2f870651675ed"}
Sep 19 08:21:41.817619 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:41.817555    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9ldsl" podStartSLOduration=17.570932304 podStartE2EDuration="19.817529728s" totalImagesPullingTime="2.246597424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:21:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:41.815532952 +0000 UTC m=+69.497116407" watchObservedRunningTime="2026-09-19 08:21:41.817529728 +0000 UTC m=+69.499113163"
Sep 19 08:21:43.813753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:43.813482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kdt58" event={"ID":"28834fd2-afab-47de-a085-52668ee84b6f","Type":"ContainerStarted","Data":"1a870da38f1fdcb9690c23bbf86219c07ff2dbf10f61451e62ec11f17e0bca33"}
Sep 19 08:21:43.813753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:43.813603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kdt58" event={"ID":"28834fd2-afab-47de-a085-52668ee84b6f","Type":"ContainerStarted","Data":"ce64be8895988c2e9842f5fe83d9157695c7c0486b4155fcdb11cff0cd746299"}
Sep 19 08:21:43.813753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:43.813642    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:43.815953 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:43.815923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kwpz5" event={"ID":"5cbb75e1-d58d-41d4-ba14-9e3194e793f8","Type":"ContainerStarted","Data":"f093e58d74d1d8fb1d15514465bff45fd5a15cc2bb253e8bf7bffe49d04cf57a"}
Sep 19 08:21:43.816080 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:43.815974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kwpz5" event={"ID":"5cbb75e1-d58d-41d4-ba14-9e3194e793f8","Type":"ContainerStarted","Data":"7dda1e34812f5220301afd2392a00d7d0dd6751104d77791632729b45176e2db"}
Sep 19 08:21:43.825076 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:43.824772    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57d7bbbb77-r92gn"]
Sep 19 08:21:43.855434 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:43.855310    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kdt58" podStartSLOduration=34.412603186 podStartE2EDuration="36.855286815s" totalImagesPullingTime="2.442683629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:43.852462899 +0000 UTC m=+71.534046334" watchObservedRunningTime="2026-09-19 08:21:43.855286815 +0000 UTC m=+71.536870250"
Sep 19 08:21:43.931615 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:43.931548    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kwpz5" podStartSLOduration=67.96250242 podStartE2EDuration="1m9.931527984s" totalImagesPullingTime="1.969025564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:43.930481115 +0000 UTC m=+71.612064547" watchObservedRunningTime="2026-09-19 08:21:43.931527984 +0000 UTC m=+71.613111418"
Sep 19 08:21:46.832783 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:46.832743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p74dm" event={"ID":"ba99474c-b9e8-42c3-8e5d-e899124b24a3","Type":"ContainerStarted","Data":"655c4750a2c206e7135876cc904c909b2874f90be895a18619997a1c836a0d28"}
Sep 19 08:21:46.833212 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:46.832872    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:21:46.834202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:46.834180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gqvcc" event={"ID":"95d065d2-7e01-4e45-83a6-22e2646d868f","Type":"ContainerStarted","Data":"cafaaaae69ead40c5c8446746e592ed2007b235f43384b2c30abadf79a739f74"}
Sep 19 08:21:46.854350 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:46.854295    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p74dm" podStartSLOduration=67.789951374 podStartE2EDuration="1m12.854277338s" totalImagesPullingTime="5.064325964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:46.852964359 +0000 UTC m=+74.534547820" watchObservedRunningTime="2026-09-19 08:21:46.854277338 +0000 UTC m=+74.535860770"
Sep 19 08:21:46.872012 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:46.871959    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gqvcc" podStartSLOduration=67.802644491 podStartE2EDuration="1m12.871941625s" totalImagesPullingTime="5.069297134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:46.871033328 +0000 UTC m=+74.552616762" watchObservedRunningTime="2026-09-19 08:21:46.871941625 +0000 UTC m=+74.553525055"
Sep 19 08:21:48.210110 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.210063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:48.212472 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.212449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e873fa1c-aba8-41f9-9f87-66b3b4439306-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-ptbj8\" (UID: \"e873fa1c-aba8-41f9-9f87-66b3b4439306\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:48.310826 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.310784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:48.310826 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.310826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:48.311457 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.311437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddfd0060-8f85-422c-85c1-9570edc61e0d-service-ca-bundle\") pod \"router-default-5f9d67fdf8-x2dbm\" (UID: \"ddfd0060-8f85-422c-85c1-9570edc61e0d\") " pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:48.313224 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.313200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-xw5l6\" (UID: \"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:48.392182 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.392147    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rd6k8\""
Sep 19 08:21:48.400436 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.400411    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"
Sep 19 08:21:48.489824 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.489745    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wkjxz\""
Sep 19 08:21:48.498045 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.498021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"
Sep 19 08:21:48.510101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.510077    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lwjc6\""
Sep 19 08:21:48.518298 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.518272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:48.670404 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.670356    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8"]
Sep 19 08:21:48.673215 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:48.673185    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode873fa1c_aba8_41f9_9f87_66b3b4439306.slice/crio-f48f72b69c2786b86731481ae87aecbb5601bbbcec1558036afccc00d3c9ef59 WatchSource:0}: Error finding container f48f72b69c2786b86731481ae87aecbb5601bbbcec1558036afccc00d3c9ef59: Status 404 returned error can't find the container with id f48f72b69c2786b86731481ae87aecbb5601bbbcec1558036afccc00d3c9ef59
Sep 19 08:21:48.768608 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.768562    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6"]
Sep 19 08:21:48.771179 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:48.771150    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode616fb2c_ac98_4a0b_8fcc_4f0b29fbcb36.slice/crio-0ae7ec2d77b77b9ded5adefbb9c222ab03d9bb3106f858212c0f2e9e8d190770 WatchSource:0}: Error finding container 0ae7ec2d77b77b9ded5adefbb9c222ab03d9bb3106f858212c0f2e9e8d190770: Status 404 returned error can't find the container with id 0ae7ec2d77b77b9ded5adefbb9c222ab03d9bb3106f858212c0f2e9e8d190770
Sep 19 08:21:48.798282 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.798223    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f9d67fdf8-x2dbm"]
Sep 19 08:21:48.800699 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:21:48.800670    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfd0060_8f85_422c_85c1_9570edc61e0d.slice/crio-b82b188265bfe0e345f8f6c5f4b7766e920d4925887aaaa6ad8dac2ff6557a3e WatchSource:0}: Error finding container b82b188265bfe0e345f8f6c5f4b7766e920d4925887aaaa6ad8dac2ff6557a3e: Status 404 returned error can't find the container with id b82b188265bfe0e345f8f6c5f4b7766e920d4925887aaaa6ad8dac2ff6557a3e
Sep 19 08:21:48.843114 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.843086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6" event={"ID":"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36","Type":"ContainerStarted","Data":"0ae7ec2d77b77b9ded5adefbb9c222ab03d9bb3106f858212c0f2e9e8d190770"}
Sep 19 08:21:48.844237 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.844213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm" event={"ID":"ddfd0060-8f85-422c-85c1-9570edc61e0d","Type":"ContainerStarted","Data":"b82b188265bfe0e345f8f6c5f4b7766e920d4925887aaaa6ad8dac2ff6557a3e"}
Sep 19 08:21:48.845278 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:48.845256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8" event={"ID":"e873fa1c-aba8-41f9-9f87-66b3b4439306","Type":"ContainerStarted","Data":"f48f72b69c2786b86731481ae87aecbb5601bbbcec1558036afccc00d3c9ef59"}
Sep 19 08:21:49.853448 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:49.853406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm" event={"ID":"ddfd0060-8f85-422c-85c1-9570edc61e0d","Type":"ContainerStarted","Data":"91f5c26ffbcabd041eb43baa9e5e89e4949116aa9181d699a11d5f93384272f6"}
Sep 19 08:21:49.855283 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:49.855245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8" event={"ID":"e873fa1c-aba8-41f9-9f87-66b3b4439306","Type":"ContainerStarted","Data":"3cb34019046f851640f0f00a2ebf4cb6d33cc79faf5f1d3da65883fd443123a1"}
Sep 19 08:21:49.877315 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:49.877262    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm" podStartSLOduration=33.877244501 podStartE2EDuration="33.877244501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:49.87594111 +0000 UTC m=+77.557524545" watchObservedRunningTime="2026-09-19 08:21:49.877244501 +0000 UTC m=+77.558827934"
Sep 19 08:21:49.892403 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:49.892334    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-ptbj8" podStartSLOduration=32.859242373 podStartE2EDuration="33.892319411s" totalImagesPullingTime="1.033077038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:49.891251474 +0000 UTC m=+77.572834910" watchObservedRunningTime="2026-09-19 08:21:49.892319411 +0000 UTC m=+77.573902843"
Sep 19 08:21:50.518574 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:50.518530    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:50.521134 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:50.521111    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:50.860555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:50.860523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6" event={"ID":"e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36","Type":"ContainerStarted","Data":"c31e25eaf253da7a5a0cf0d1be32cc6b9febf0fce88648d712ed7df4b0214ad0"}
Sep 19 08:21:50.860931 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:50.860790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:50.861986 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:50.861963    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f9d67fdf8-x2dbm"
Sep 19 08:21:50.878296 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:50.878028    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-xw5l6" podStartSLOduration=32.870519766 podStartE2EDuration="34.878006786s" totalImagesPullingTime="2.00748702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:50.87791539 +0000 UTC m=+78.559498835" watchObservedRunningTime="2026-09-19 08:21:50.878006786 +0000 UTC m=+78.559590222"
Sep 19 08:21:53.822093 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:53.822058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kdt58"
Sep 19 08:21:53.829690 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:21:53.829663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:22:08.850643 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:08.850599    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn" podUID="d0e6b508-19eb-4062-afd9-ad5260f3f153" containerName="registry" containerID="cri-o://809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028" gracePeriod=30
Sep 19 08:22:09.083752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.083728    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:22:09.257923 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.257817    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-certificates\") pod \"d0e6b508-19eb-4062-afd9-ad5260f3f153\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") "
Sep 19 08:22:09.257923 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.257885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0e6b508-19eb-4062-afd9-ad5260f3f153-ca-trust-extracted\") pod \"d0e6b508-19eb-4062-afd9-ad5260f3f153\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") "
Sep 19 08:22:09.257923 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.257913    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2b8j\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-kube-api-access-h2b8j\") pod \"d0e6b508-19eb-4062-afd9-ad5260f3f153\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") "
Sep 19 08:22:09.258203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.258021    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-installation-pull-secrets\") pod \"d0e6b508-19eb-4062-afd9-ad5260f3f153\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") "
Sep 19 08:22:09.258203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.258097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-trusted-ca\") pod \"d0e6b508-19eb-4062-afd9-ad5260f3f153\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") "
Sep 19 08:22:09.258203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.258148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-image-registry-private-configuration\") pod \"d0e6b508-19eb-4062-afd9-ad5260f3f153\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") "
Sep 19 08:22:09.258203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.258182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") pod \"d0e6b508-19eb-4062-afd9-ad5260f3f153\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") "
Sep 19 08:22:09.258437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.258211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-bound-sa-token\") pod \"d0e6b508-19eb-4062-afd9-ad5260f3f153\" (UID: \"d0e6b508-19eb-4062-afd9-ad5260f3f153\") "
Sep 19 08:22:09.258505 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.258306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d0e6b508-19eb-4062-afd9-ad5260f3f153" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:09.258629 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.258596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d0e6b508-19eb-4062-afd9-ad5260f3f153" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:09.260510 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.260480    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-kube-api-access-h2b8j" (OuterVolumeSpecName: "kube-api-access-h2b8j") pod "d0e6b508-19eb-4062-afd9-ad5260f3f153" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153"). InnerVolumeSpecName "kube-api-access-h2b8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:22:09.260637 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.260484    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d0e6b508-19eb-4062-afd9-ad5260f3f153" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:09.260756 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.260732    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d0e6b508-19eb-4062-afd9-ad5260f3f153" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:09.260851 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.260829    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d0e6b508-19eb-4062-afd9-ad5260f3f153" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:22:09.260947 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.260928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d0e6b508-19eb-4062-afd9-ad5260f3f153" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:22:09.266719 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.266691    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e6b508-19eb-4062-afd9-ad5260f3f153-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d0e6b508-19eb-4062-afd9-ad5260f3f153" (UID: "d0e6b508-19eb-4062-afd9-ad5260f3f153"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:22:09.359545 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.359503    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-image-registry-private-configuration\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:22:09.359545 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.359539    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-tls\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:22:09.359545 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.359555    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-bound-sa-token\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:22:09.359796 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.359566    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-registry-certificates\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:22:09.359796 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.359578    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d0e6b508-19eb-4062-afd9-ad5260f3f153-ca-trust-extracted\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:22:09.359796 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.359590    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2b8j\" (UniqueName: \"kubernetes.io/projected/d0e6b508-19eb-4062-afd9-ad5260f3f153-kube-api-access-h2b8j\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:22:09.359796 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.359601    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d0e6b508-19eb-4062-afd9-ad5260f3f153-installation-pull-secrets\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:22:09.359796 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.359612    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0e6b508-19eb-4062-afd9-ad5260f3f153-trusted-ca\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:22:09.928491 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.928455    2577 generic.go:356] "Generic (PLEG): container finished" podID="d0e6b508-19eb-4062-afd9-ad5260f3f153" containerID="809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028" exitCode=0
Sep 19 08:22:09.928925 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.928519    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn"
Sep 19 08:22:09.928925 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.928543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn" event={"ID":"d0e6b508-19eb-4062-afd9-ad5260f3f153","Type":"ContainerDied","Data":"809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028"}
Sep 19 08:22:09.928925 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.928592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7bbbb77-r92gn" event={"ID":"d0e6b508-19eb-4062-afd9-ad5260f3f153","Type":"ContainerDied","Data":"6759e325b7da221aa47c460fe9245b8aca11d2ea93668347f1f614c946115ca4"}
Sep 19 08:22:09.928925 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.928613    2577 scope.go:117] "RemoveContainer" containerID="809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028"
Sep 19 08:22:09.937727 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.937564    2577 scope.go:117] "RemoveContainer" containerID="809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028"
Sep 19 08:22:09.937934 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:22:09.937912    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028\": container with ID starting with 809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028 not found: ID does not exist" containerID="809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028"
Sep 19 08:22:09.938007 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.937947    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028"} err="failed to get container status \"809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028\": rpc error: code = NotFound desc = could not find container \"809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028\": container with ID starting with 809a70b6bb50a5166baeed9f8b57d951bdac75845255e6742626a75347b4a028 not found: ID does not exist"
Sep 19 08:22:09.950358 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.950329    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57d7bbbb77-r92gn"]
Sep 19 08:22:09.955770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:09.955742    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57d7bbbb77-r92gn"]
Sep 19 08:22:10.502282 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:10.502248    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e6b508-19eb-4062-afd9-ad5260f3f153" path="/var/lib/kubelet/pods/d0e6b508-19eb-4062-afd9-ad5260f3f153/volumes"
Sep 19 08:22:17.751646 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.751607    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"]
Sep 19 08:22:17.752103 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.751846    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e6b508-19eb-4062-afd9-ad5260f3f153" containerName="registry"
Sep 19 08:22:17.752103 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.751857    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e6b508-19eb-4062-afd9-ad5260f3f153" containerName="registry"
Sep 19 08:22:17.752103 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.751903    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e6b508-19eb-4062-afd9-ad5260f3f153" containerName="registry"
Sep 19 08:22:17.754803 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.754787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:17.762263 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.762241    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5v29x\""
Sep 19 08:22:17.762263 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.762260    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 08:22:17.762488 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.762242    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 19 08:22:17.762488 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.762242    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 19 08:22:17.772101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.772073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"]
Sep 19 08:22:17.787771 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.787740    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-dkd42"]
Sep 19 08:22:17.790956 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.790941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:17.793840 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.793821    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 19 08:22:17.794134 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.794113    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hcgcr\""
Sep 19 08:22:17.794249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.794229    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 19 08:22:17.794298 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.794260    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 19 08:22:17.806051 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.806023    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-dkd42"]
Sep 19 08:22:17.813303 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.813275    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qbs7v"]
Sep 19 08:22:17.816489 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.816473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.818818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.818796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 08:22:17.819242 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819224    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jtrxn\""
Sep 19 08:22:17.819322 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819305    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 08:22:17.819617 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.819694 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819638    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 08:22:17.819694 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-textfile\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.819770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkqw\" (UniqueName: \"kubernetes.io/projected/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-kube-api-access-6vkqw\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.819770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.819854 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-tls\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.819953 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-metrics-client-ca\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.819995 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-sys\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.819995 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.819984    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-root\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.820077 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.820007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-wtmp\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.840177 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.840139    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p74dm"
Sep 19 08:22:17.920831 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.920801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/251868a1-d2a6-42d9-b51a-1654c517d3d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:17.921022 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.920846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921022 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.920903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251868a1-d2a6-42d9-b51a-1654c517d3d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:17.921022 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.920938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:17.921022 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.920958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:17.921022 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.920974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxrd\" (UniqueName: \"kubernetes.io/projected/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-api-access-jpxrd\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:17.921022 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.920994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:17.921346 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-textfile\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921346 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkqw\" (UniqueName: \"kubernetes.io/projected/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-kube-api-access-6vkqw\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921346 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921346 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-tls\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921346 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251868a1-d2a6-42d9-b51a-1654c517d3d8-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:17.921346 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:17.921346 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:17.921718 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-textfile\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921718 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-metrics-client-ca\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921718 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tbf\" (UniqueName: \"kubernetes.io/projected/251868a1-d2a6-42d9-b51a-1654c517d3d8-kube-api-access-b6tbf\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:17.921718 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-sys\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921718 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-root\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921718 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-sys\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.921718 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921691    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-wtmp\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.922073 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.922073 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-root\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.922073 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-wtmp\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.922073 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.921901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-metrics-client-ca\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.923507 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.923480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.923633 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.923614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-node-exporter-tls\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:17.934072 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:17.934049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkqw\" (UniqueName: \"kubernetes.io/projected/e7657b45-d17a-41ae-8dc6-8e5e221aa1cf-kube-api-access-6vkqw\") pod \"node-exporter-qbs7v\" (UID: \"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf\") " pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:18.023007 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.022911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251868a1-d2a6-42d9-b51a-1654c517d3d8-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:18.023007 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.022955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.023007 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.022977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.023297 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tbf\" (UniqueName: \"kubernetes.io/projected/251868a1-d2a6-42d9-b51a-1654c517d3d8-kube-api-access-b6tbf\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:18.023297 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/251868a1-d2a6-42d9-b51a-1654c517d3d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:18.023297 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251868a1-d2a6-42d9-b51a-1654c517d3d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:18.023297 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.023297 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.023595 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxrd\" (UniqueName: \"kubernetes.io/projected/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-api-access-jpxrd\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.023595 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.023723 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.023790 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.023790 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.023766    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251868a1-d2a6-42d9-b51a-1654c517d3d8-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:18.024103 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.024075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.025729 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.025706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/251868a1-d2a6-42d9-b51a-1654c517d3d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:18.025833 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.025786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.025904 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.025857    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.026123 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.026103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251868a1-d2a6-42d9-b51a-1654c517d3d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:18.031976 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.031944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxrd\" (UniqueName: \"kubernetes.io/projected/6be1325b-dc1c-49cd-bc59-f3d0a92fbadb-kube-api-access-jpxrd\") pod \"kube-state-metrics-84c885658c-dkd42\" (UID: \"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.032072 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.032045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tbf\" (UniqueName: \"kubernetes.io/projected/251868a1-d2a6-42d9-b51a-1654c517d3d8-kube-api-access-b6tbf\") pod \"openshift-state-metrics-86c5bc79d6-7ftpc\" (UID: \"251868a1-d2a6-42d9-b51a-1654c517d3d8\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:18.063840 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.063803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"
Sep 19 08:22:18.100412 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.100358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42"
Sep 19 08:22:18.126095 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.126030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qbs7v"
Sep 19 08:22:18.140134 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:22:18.140105    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7657b45_d17a_41ae_8dc6_8e5e221aa1cf.slice/crio-37b820531423339967c3ff5d9332530a29662a7ce2cdcb30b14ff845b73bdb64 WatchSource:0}: Error finding container 37b820531423339967c3ff5d9332530a29662a7ce2cdcb30b14ff845b73bdb64: Status 404 returned error can't find the container with id 37b820531423339967c3ff5d9332530a29662a7ce2cdcb30b14ff845b73bdb64
Sep 19 08:22:18.343190 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.343161    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc"]
Sep 19 08:22:18.344978 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:22:18.344947    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251868a1_d2a6_42d9_b51a_1654c517d3d8.slice/crio-18cbeb6d9291d6044ecc7f0e0641d650d94c64090fb7c71f9760374e8ba3b8d3 WatchSource:0}: Error finding container 18cbeb6d9291d6044ecc7f0e0641d650d94c64090fb7c71f9760374e8ba3b8d3: Status 404 returned error can't find the container with id 18cbeb6d9291d6044ecc7f0e0641d650d94c64090fb7c71f9760374e8ba3b8d3
Sep 19 08:22:18.383017 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.382993    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-dkd42"]
Sep 19 08:22:18.385511 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:22:18.385481    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be1325b_dc1c_49cd_bc59_f3d0a92fbadb.slice/crio-5e36c2faf584f2272b8223194cf61430c2fbb9c14cf8bae42d8a663118d86d81 WatchSource:0}: Error finding container 5e36c2faf584f2272b8223194cf61430c2fbb9c14cf8bae42d8a663118d86d81: Status 404 returned error can't find the container with id 5e36c2faf584f2272b8223194cf61430c2fbb9c14cf8bae42d8a663118d86d81
Sep 19 08:22:18.964071 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.963991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbs7v" event={"ID":"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf","Type":"ContainerStarted","Data":"37b820531423339967c3ff5d9332530a29662a7ce2cdcb30b14ff845b73bdb64"}
Sep 19 08:22:18.966596 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.966569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc" event={"ID":"251868a1-d2a6-42d9-b51a-1654c517d3d8","Type":"ContainerStarted","Data":"43f76ca7ee42a3d98caa67905242522253a49949a23949c05079239894f40c56"}
Sep 19 08:22:18.966742 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.966628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc" event={"ID":"251868a1-d2a6-42d9-b51a-1654c517d3d8","Type":"ContainerStarted","Data":"44d3ec55e12b9723259375f82a5cd57229613c16db710ba0b02ce7a4a51e0f01"}
Sep 19 08:22:18.966742 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.966656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc" event={"ID":"251868a1-d2a6-42d9-b51a-1654c517d3d8","Type":"ContainerStarted","Data":"18cbeb6d9291d6044ecc7f0e0641d650d94c64090fb7c71f9760374e8ba3b8d3"}
Sep 19 08:22:18.967842 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:18.967810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42" event={"ID":"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb","Type":"ContainerStarted","Data":"5e36c2faf584f2272b8223194cf61430c2fbb9c14cf8bae42d8a663118d86d81"}
Sep 19 08:22:20.977013 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:20.976972    2577 generic.go:356] "Generic (PLEG): container finished" podID="e7657b45-d17a-41ae-8dc6-8e5e221aa1cf" containerID="85ff4294a71a59b45670f5f67e2560cb24660b5633e7c0e29ed2619ec585bc91" exitCode=0
Sep 19 08:22:20.977422 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:20.977053    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbs7v" event={"ID":"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf","Type":"ContainerDied","Data":"85ff4294a71a59b45670f5f67e2560cb24660b5633e7c0e29ed2619ec585bc91"}
Sep 19 08:22:20.979541 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:20.979515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc" event={"ID":"251868a1-d2a6-42d9-b51a-1654c517d3d8","Type":"ContainerStarted","Data":"d014389df9fbb014b1f8017cbc4210d84da2adccec2e58fd7c7e3b7968c77567"}
Sep 19 08:22:20.981575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:20.981555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42" event={"ID":"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb","Type":"ContainerStarted","Data":"8cdd02eade4348a1147423c1adc6e8203e137a4a57d7a159cb104a993bc4f93e"}
Sep 19 08:22:20.981655 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:20.981598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42" event={"ID":"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb","Type":"ContainerStarted","Data":"94bda88deb498663b32dee63513b6d4887a400224a27e1b65ecf6e17d1cb91b9"}
Sep 19 08:22:21.014698 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:21.014633    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-7ftpc" podStartSLOduration=1.8531234140000001 podStartE2EDuration="4.014580948s" totalImagesPullingTime="2.161457534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:21.013730069 +0000 UTC m=+108.695313504" watchObservedRunningTime="2026-09-19 08:22:21.014580948 +0000 UTC m=+108.696164383"
Sep 19 08:22:21.991278 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:21.991238    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbs7v" event={"ID":"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf","Type":"ContainerStarted","Data":"1c669110a9924a144b0b864d0d4782934ae022dd8f2989b66d934ac63ec663f2"}
Sep 19 08:22:21.991758 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:21.991332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbs7v" event={"ID":"e7657b45-d17a-41ae-8dc6-8e5e221aa1cf","Type":"ContainerStarted","Data":"384d86aa24e8565d72fd6172fdce5a5f0f4e9d2bfa8cd4ac02e996298efed592"}
Sep 19 08:22:21.993418 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:21.993393    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42" event={"ID":"6be1325b-dc1c-49cd-bc59-f3d0a92fbadb","Type":"ContainerStarted","Data":"786f9a5ed5e9475584dd8b7b674c28992858d6868b4a75aaf2887fd3c9546d17"}
Sep 19 08:22:22.030579 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.030419    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qbs7v" podStartSLOduration=2.427920274 podStartE2EDuration="5.030398179s" totalImagesPullingTime="2.512884717s" totalInitContainerRuntime="89.593188ms" isStatefulPod=true podCreationTimestamp="2026-09-19 08:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:22.010563151 +0000 UTC m=+109.692146584" watchObservedRunningTime="2026-09-19 08:22:22.030398179 +0000 UTC m=+109.711981613"
Sep 19 08:22:22.031137 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.031106    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-dkd42" podStartSLOduration=2.76258277 podStartE2EDuration="5.031097317s" totalImagesPullingTime="2.268514547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:22.030076777 +0000 UTC m=+109.711660286" watchObservedRunningTime="2026-09-19 08:22:22.031097317 +0000 UTC m=+109.712680750"
Sep 19 08:22:22.279753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.279671    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b996944bf-762gj"]
Sep 19 08:22:22.311437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.311404    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b996944bf-762gj"]
Sep 19 08:22:22.311574 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.311469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.314162 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.314142    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 19 08:22:22.314162 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.314157    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 19 08:22:22.315164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.315146    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 19 08:22:22.315241 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.315195    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 19 08:22:22.315241 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.315234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ffqlvm46n1169\""
Sep 19 08:22:22.315332 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.315237    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-7sx84\""
Sep 19 08:22:22.353773 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.353745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmmb\" (UniqueName: \"kubernetes.io/projected/3c5550a3-16b8-4853-823e-619127563a54-kube-api-access-wlmmb\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.353773 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.353777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3c5550a3-16b8-4853-823e-619127563a54-audit-log\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.354019 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.353798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3c5550a3-16b8-4853-823e-619127563a54-client-ca-bundle\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.354019 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.353816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3c5550a3-16b8-4853-823e-619127563a54-secret-metrics-server-tls\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.354019 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.353893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c5550a3-16b8-4853-823e-619127563a54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.354019 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.353920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3c5550a3-16b8-4853-823e-619127563a54-secret-metrics-server-client-certs\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.354019 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.353970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3c5550a3-16b8-4853-823e-619127563a54-metrics-server-audit-profiles\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.454805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.454763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmmb\" (UniqueName: \"kubernetes.io/projected/3c5550a3-16b8-4853-823e-619127563a54-kube-api-access-wlmmb\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.454969 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.454813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3c5550a3-16b8-4853-823e-619127563a54-audit-log\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.454969 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.454855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3c5550a3-16b8-4853-823e-619127563a54-client-ca-bundle\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.454969 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.454884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3c5550a3-16b8-4853-823e-619127563a54-secret-metrics-server-tls\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.454969 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.454944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c5550a3-16b8-4853-823e-619127563a54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.455199 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.454969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3c5550a3-16b8-4853-823e-619127563a54-secret-metrics-server-client-certs\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.455199 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.455009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3c5550a3-16b8-4853-823e-619127563a54-metrics-server-audit-profiles\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.455485 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.455456    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3c5550a3-16b8-4853-823e-619127563a54-audit-log\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.455837 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.455816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c5550a3-16b8-4853-823e-619127563a54-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.456428 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.456405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3c5550a3-16b8-4853-823e-619127563a54-metrics-server-audit-profiles\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.457685 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.457661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3c5550a3-16b8-4853-823e-619127563a54-client-ca-bundle\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.457786 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.457704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3c5550a3-16b8-4853-823e-619127563a54-secret-metrics-server-client-certs\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.457842 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.457816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3c5550a3-16b8-4853-823e-619127563a54-secret-metrics-server-tls\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.463307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.463284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmmb\" (UniqueName: \"kubernetes.io/projected/3c5550a3-16b8-4853-823e-619127563a54-kube-api-access-wlmmb\") pod \"metrics-server-6b996944bf-762gj\" (UID: \"3c5550a3-16b8-4853-823e-619127563a54\") " pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.621129 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.621096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:22.925202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.925124    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b996944bf-762gj"]
Sep 19 08:22:22.928479 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:22:22.928448    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5550a3_16b8_4853_823e_619127563a54.slice/crio-9a164f3fd96c328dd341d828e19cc1021b61350e686b4dd79ef5e022d1557a59 WatchSource:0}: Error finding container 9a164f3fd96c328dd341d828e19cc1021b61350e686b4dd79ef5e022d1557a59: Status 404 returned error can't find the container with id 9a164f3fd96c328dd341d828e19cc1021b61350e686b4dd79ef5e022d1557a59
Sep 19 08:22:22.997828 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:22.997790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b996944bf-762gj" event={"ID":"3c5550a3-16b8-4853-823e-619127563a54","Type":"ContainerStarted","Data":"9a164f3fd96c328dd341d828e19cc1021b61350e686b4dd79ef5e022d1557a59"}
Sep 19 08:22:25.009464 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:25.009423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b996944bf-762gj" event={"ID":"3c5550a3-16b8-4853-823e-619127563a54","Type":"ContainerStarted","Data":"8642eaaaef07672621b222e9465e71692305b11c73479bfe8fea61bbeebb2d0b"}
Sep 19 08:22:25.033594 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:25.033540    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b996944bf-762gj" podStartSLOduration=1.646399065 podStartE2EDuration="3.033524724s" totalImagesPullingTime="1.387125659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:25.03243191 +0000 UTC m=+112.714015335" watchObservedRunningTime="2026-09-19 08:22:25.033524724 +0000 UTC m=+112.715108157"
Sep 19 08:22:42.621725 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:42.621674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:42.622239 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:42.621758    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:22:45.082718 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:45.082683    2577 generic.go:356] "Generic (PLEG): container finished" podID="dde16ec8-2ee3-48a3-a755-b8d40692076b" containerID="41c60a10396da3f5f1038c8f311b3c9b2e8c9e56c3f920643dbc97e2f1a73335" exitCode=0
Sep 19 08:22:45.083096 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:45.082756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml" event={"ID":"dde16ec8-2ee3-48a3-a755-b8d40692076b","Type":"ContainerDied","Data":"41c60a10396da3f5f1038c8f311b3c9b2e8c9e56c3f920643dbc97e2f1a73335"}
Sep 19 08:22:45.083096 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:45.083069    2577 scope.go:117] "RemoveContainer" containerID="41c60a10396da3f5f1038c8f311b3c9b2e8c9e56c3f920643dbc97e2f1a73335"
Sep 19 08:22:46.087666 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:46.087633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-48wml" event={"ID":"dde16ec8-2ee3-48a3-a755-b8d40692076b","Type":"ContainerStarted","Data":"87c598154dc1301cacd26d81d44dc14e8c8ae3b779376124b73c2cbf2de63505"}
Sep 19 08:22:55.120307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:55.120270    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a7f5c37-2015-4424-b262-b8a1729653e2" containerID="4cad9a304ed82311bc324aea2acc093acd254c7a6bd4a8f1113b53c2528d868d" exitCode=0
Sep 19 08:22:55.120751 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:55.120341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq" event={"ID":"1a7f5c37-2015-4424-b262-b8a1729653e2","Type":"ContainerDied","Data":"4cad9a304ed82311bc324aea2acc093acd254c7a6bd4a8f1113b53c2528d868d"}
Sep 19 08:22:55.120751 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:55.120693    2577 scope.go:117] "RemoveContainer" containerID="4cad9a304ed82311bc324aea2acc093acd254c7a6bd4a8f1113b53c2528d868d"
Sep 19 08:22:56.125242 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:22:56.125210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6q6kq" event={"ID":"1a7f5c37-2015-4424-b262-b8a1729653e2","Type":"ContainerStarted","Data":"1cdd16ded48e8b79a5e03f9139282efd50ad4fc5b6832ab40d86771db136cde7"}
Sep 19 08:23:02.627381 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:23:02.627334    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:23:02.631150 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:23:02.631118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b996944bf-762gj"
Sep 19 08:25:34.408965 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:34.408932    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:25:34.410992 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:34.410967    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:25:34.414193 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:34.414155    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 08:25:37.130490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.130455    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"]
Sep 19 08:25:37.133628 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.133612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"
Sep 19 08:25:37.137311 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.137286    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 19 08:25:37.137462 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.137293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 08:25:37.137462 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.137335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 08:25:37.137462 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.137355    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 08:25:37.137462 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.137390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-s99c8\""
Sep 19 08:25:37.143928 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.143909    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"]
Sep 19 08:25:37.243257 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.243228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd753516-83d5-40e3-aea5-7ca21d08d588-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-556dd7645d-5rtlv\" (UID: \"cd753516-83d5-40e3-aea5-7ca21d08d588\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"
Sep 19 08:25:37.243453 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.243280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h558v\" (UniqueName: \"kubernetes.io/projected/cd753516-83d5-40e3-aea5-7ca21d08d588-kube-api-access-h558v\") pod \"managed-serviceaccount-addon-agent-556dd7645d-5rtlv\" (UID: \"cd753516-83d5-40e3-aea5-7ca21d08d588\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"
Sep 19 08:25:37.349287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.349259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h558v\" (UniqueName: \"kubernetes.io/projected/cd753516-83d5-40e3-aea5-7ca21d08d588-kube-api-access-h558v\") pod \"managed-serviceaccount-addon-agent-556dd7645d-5rtlv\" (UID: \"cd753516-83d5-40e3-aea5-7ca21d08d588\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"
Sep 19 08:25:37.349475 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.349343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd753516-83d5-40e3-aea5-7ca21d08d588-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-556dd7645d-5rtlv\" (UID: \"cd753516-83d5-40e3-aea5-7ca21d08d588\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"
Sep 19 08:25:37.351863 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.351834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd753516-83d5-40e3-aea5-7ca21d08d588-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-556dd7645d-5rtlv\" (UID: \"cd753516-83d5-40e3-aea5-7ca21d08d588\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"
Sep 19 08:25:37.357520 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.357499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h558v\" (UniqueName: \"kubernetes.io/projected/cd753516-83d5-40e3-aea5-7ca21d08d588-kube-api-access-h558v\") pod \"managed-serviceaccount-addon-agent-556dd7645d-5rtlv\" (UID: \"cd753516-83d5-40e3-aea5-7ca21d08d588\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"
Sep 19 08:25:37.457696 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.457603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"
Sep 19 08:25:37.723581 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.723496    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv"]
Sep 19 08:25:37.726249 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:25:37.726221    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd753516_83d5_40e3_aea5_7ca21d08d588.slice/crio-1c9f4c259e76e8a8fe53e9be2253419d7f6c1ddf36c5c40b58ce620e2525a4f5 WatchSource:0}: Error finding container 1c9f4c259e76e8a8fe53e9be2253419d7f6c1ddf36c5c40b58ce620e2525a4f5: Status 404 returned error can't find the container with id 1c9f4c259e76e8a8fe53e9be2253419d7f6c1ddf36c5c40b58ce620e2525a4f5
Sep 19 08:25:37.727995 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:37.727978    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:25:38.709542 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:38.709501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv" event={"ID":"cd753516-83d5-40e3-aea5-7ca21d08d588","Type":"ContainerStarted","Data":"1c9f4c259e76e8a8fe53e9be2253419d7f6c1ddf36c5c40b58ce620e2525a4f5"}
Sep 19 08:25:40.717981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:40.717953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv" event={"ID":"cd753516-83d5-40e3-aea5-7ca21d08d588","Type":"ContainerStarted","Data":"fedb14338ce18593a0d0b4943a5ddb26b824141860134fbd71c16c54cc982c35"}
Sep 19 08:25:40.732900 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:25:40.732855    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-556dd7645d-5rtlv" podStartSLOduration=1.340623392 podStartE2EDuration="3.732840496s" totalImagesPullingTime="2.392217104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:40.732169354 +0000 UTC m=+308.413752787" watchObservedRunningTime="2026-09-19 08:25:40.732840496 +0000 UTC m=+308.414423926"
Sep 19 08:26:26.178615 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.178536    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"]
Sep 19 08:26:26.181824 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.181805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.184228 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.184203    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 19 08:26:26.184333 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.184270    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7lq9v\""
Sep 19 08:26:26.185044 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.185024    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 19 08:26:26.191658 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.191635    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"]
Sep 19 08:26:26.303002 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.302963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.303206 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.303023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.303206 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.303094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bsqb\" (UniqueName: \"kubernetes.io/projected/941eee57-b7a6-4b57-97fc-eb2093eb500d-kube-api-access-7bsqb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.404030 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.403997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.404142 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.404053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.404142 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.404085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bsqb\" (UniqueName: \"kubernetes.io/projected/941eee57-b7a6-4b57-97fc-eb2093eb500d-kube-api-access-7bsqb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.404472 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.404454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.404516 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.404474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.412502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.412483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bsqb\" (UniqueName: \"kubernetes.io/projected/941eee57-b7a6-4b57-97fc-eb2093eb500d-kube-api-access-7bsqb\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.491108 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.491020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:26.763543 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.763508    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"]
Sep 19 08:26:26.765751 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:26:26.765727    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941eee57_b7a6_4b57_97fc_eb2093eb500d.slice/crio-0db26061284a867670061b7cad88dcde3253003227a38b69c23e32b0e886bf28 WatchSource:0}: Error finding container 0db26061284a867670061b7cad88dcde3253003227a38b69c23e32b0e886bf28: Status 404 returned error can't find the container with id 0db26061284a867670061b7cad88dcde3253003227a38b69c23e32b0e886bf28
Sep 19 08:26:26.884795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:26.884760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v" event={"ID":"941eee57-b7a6-4b57-97fc-eb2093eb500d","Type":"ContainerStarted","Data":"0db26061284a867670061b7cad88dcde3253003227a38b69c23e32b0e886bf28"}
Sep 19 08:26:32.908747 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:32.908709    2577 generic.go:356] "Generic (PLEG): container finished" podID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerID="c052f49bb30c16b4599dac1110c3947c09953ab93e7824402481f19e0f470368" exitCode=0
Sep 19 08:26:32.909163 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:32.908761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v" event={"ID":"941eee57-b7a6-4b57-97fc-eb2093eb500d","Type":"ContainerDied","Data":"c052f49bb30c16b4599dac1110c3947c09953ab93e7824402481f19e0f470368"}
Sep 19 08:26:35.924103 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:35.924065    2577 generic.go:356] "Generic (PLEG): container finished" podID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerID="c03085ff46b2fb31180ca9e1d6f5b58e4fff74a6e631636ddda0278b5a4016da" exitCode=0
Sep 19 08:26:35.924527 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:35.924121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v" event={"ID":"941eee57-b7a6-4b57-97fc-eb2093eb500d","Type":"ContainerDied","Data":"c03085ff46b2fb31180ca9e1d6f5b58e4fff74a6e631636ddda0278b5a4016da"}
Sep 19 08:26:42.957101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:42.957068    2577 generic.go:356] "Generic (PLEG): container finished" podID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerID="59fe59f3d966fbc59b73b062f68a10ae7e936aeaacbd9e72538e9eb33aaadcc5" exitCode=0
Sep 19 08:26:42.957583 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:42.957122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v" event={"ID":"941eee57-b7a6-4b57-97fc-eb2093eb500d","Type":"ContainerDied","Data":"59fe59f3d966fbc59b73b062f68a10ae7e936aeaacbd9e72538e9eb33aaadcc5"}
Sep 19 08:26:44.080600 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.080574    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:44.253166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.253059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-bundle\") pod \"941eee57-b7a6-4b57-97fc-eb2093eb500d\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") "
Sep 19 08:26:44.253166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.253159    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bsqb\" (UniqueName: \"kubernetes.io/projected/941eee57-b7a6-4b57-97fc-eb2093eb500d-kube-api-access-7bsqb\") pod \"941eee57-b7a6-4b57-97fc-eb2093eb500d\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") "
Sep 19 08:26:44.253422 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.253228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-util\") pod \"941eee57-b7a6-4b57-97fc-eb2093eb500d\" (UID: \"941eee57-b7a6-4b57-97fc-eb2093eb500d\") "
Sep 19 08:26:44.253904 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.253877    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-bundle" (OuterVolumeSpecName: "bundle") pod "941eee57-b7a6-4b57-97fc-eb2093eb500d" (UID: "941eee57-b7a6-4b57-97fc-eb2093eb500d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:26:44.255395 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.255353    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941eee57-b7a6-4b57-97fc-eb2093eb500d-kube-api-access-7bsqb" (OuterVolumeSpecName: "kube-api-access-7bsqb") pod "941eee57-b7a6-4b57-97fc-eb2093eb500d" (UID: "941eee57-b7a6-4b57-97fc-eb2093eb500d"). InnerVolumeSpecName "kube-api-access-7bsqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:26:44.258085 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.258056    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-util" (OuterVolumeSpecName: "util") pod "941eee57-b7a6-4b57-97fc-eb2093eb500d" (UID: "941eee57-b7a6-4b57-97fc-eb2093eb500d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:26:44.354501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.354465    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-util\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:26:44.354501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.354494    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/941eee57-b7a6-4b57-97fc-eb2093eb500d-bundle\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:26:44.354501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.354503    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bsqb\" (UniqueName: \"kubernetes.io/projected/941eee57-b7a6-4b57-97fc-eb2093eb500d-kube-api-access-7bsqb\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:26:44.966506 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.966471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v" event={"ID":"941eee57-b7a6-4b57-97fc-eb2093eb500d","Type":"ContainerDied","Data":"0db26061284a867670061b7cad88dcde3253003227a38b69c23e32b0e886bf28"}
Sep 19 08:26:44.966506 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.966502    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2xl8v"
Sep 19 08:26:44.966700 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:44.966519    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0db26061284a867670061b7cad88dcde3253003227a38b69c23e32b0e886bf28"
Sep 19 08:26:48.754047 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.754013    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"]
Sep 19 08:26:48.754451 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.754287    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerName="util"
Sep 19 08:26:48.754451 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.754297    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerName="util"
Sep 19 08:26:48.754451 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.754308    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerName="extract"
Sep 19 08:26:48.754451 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.754313    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerName="extract"
Sep 19 08:26:48.754451 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.754323    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerName="pull"
Sep 19 08:26:48.754451 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.754328    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerName="pull"
Sep 19 08:26:48.754451 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.754395    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="941eee57-b7a6-4b57-97fc-eb2093eb500d" containerName="extract"
Sep 19 08:26:48.796257 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.796234    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"]
Sep 19 08:26:48.796415 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.796324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:26:48.799373 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.799338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-dpjx8\""
Sep 19 08:26:48.799617 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.799598    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 19 08:26:48.799771 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.799757    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 19 08:26:48.799840 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.799767    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 19 08:26:48.887868 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.887833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzv95\" (UniqueName: \"kubernetes.io/projected/eda7dd45-6ce2-4466-b17e-6db87b769f18-kube-api-access-pzv95\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pqv7d\" (UID: \"eda7dd45-6ce2-4466-b17e-6db87b769f18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:26:48.888026 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.887877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/eda7dd45-6ce2-4466-b17e-6db87b769f18-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pqv7d\" (UID: \"eda7dd45-6ce2-4466-b17e-6db87b769f18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:26:48.989132 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.989102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzv95\" (UniqueName: \"kubernetes.io/projected/eda7dd45-6ce2-4466-b17e-6db87b769f18-kube-api-access-pzv95\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pqv7d\" (UID: \"eda7dd45-6ce2-4466-b17e-6db87b769f18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:26:48.989297 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.989147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/eda7dd45-6ce2-4466-b17e-6db87b769f18-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pqv7d\" (UID: \"eda7dd45-6ce2-4466-b17e-6db87b769f18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:26:48.991466 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.991444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/eda7dd45-6ce2-4466-b17e-6db87b769f18-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pqv7d\" (UID: \"eda7dd45-6ce2-4466-b17e-6db87b769f18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:26:48.998295 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:48.998266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzv95\" (UniqueName: \"kubernetes.io/projected/eda7dd45-6ce2-4466-b17e-6db87b769f18-kube-api-access-pzv95\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-pqv7d\" (UID: \"eda7dd45-6ce2-4466-b17e-6db87b769f18\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:26:49.106089 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:49.106056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:26:49.414624 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:49.414595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"]
Sep 19 08:26:49.417379 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:26:49.417333    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda7dd45_6ce2_4466_b17e_6db87b769f18.slice/crio-ad26d3e4b7ee153d4b91b4215975b9a027d26b04da6b7f3c5db6a307ad9acf03 WatchSource:0}: Error finding container ad26d3e4b7ee153d4b91b4215975b9a027d26b04da6b7f3c5db6a307ad9acf03: Status 404 returned error can't find the container with id ad26d3e4b7ee153d4b91b4215975b9a027d26b04da6b7f3c5db6a307ad9acf03
Sep 19 08:26:49.986271 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:49.986236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d" event={"ID":"eda7dd45-6ce2-4466-b17e-6db87b769f18","Type":"ContainerStarted","Data":"ad26d3e4b7ee153d4b91b4215975b9a027d26b04da6b7f3c5db6a307ad9acf03"}
Sep 19 08:26:53.001302 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.001267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d" event={"ID":"eda7dd45-6ce2-4466-b17e-6db87b769f18","Type":"ContainerStarted","Data":"ad988f268bb1f249681dc4a0d9b13ee6f8a629b7daa1db76c05b955833daa0ac"}
Sep 19 08:26:53.001693 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.001398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:26:53.022977 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.022895    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d" podStartSLOduration=1.684727132 podStartE2EDuration="5.022881183s" totalImagesPullingTime="3.338154051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:53.021933322 +0000 UTC m=+380.703516754" watchObservedRunningTime="2026-09-19 08:26:53.022881183 +0000 UTC m=+380.704464616"
Sep 19 08:26:53.588419 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.588357    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5sgdl"]
Sep 19 08:26:53.611288 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.611259    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5sgdl"]
Sep 19 08:26:53.611444 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.611390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:53.614073 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.614052    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 19 08:26:53.614073 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.614070    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tnjsj\""
Sep 19 08:26:53.614073 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.614053    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 19 08:26:53.730048 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.730011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-certificates\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:53.730048 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.730051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-cabundle0\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:53.730257 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.730079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m984t\" (UniqueName: \"kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-kube-api-access-m984t\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:53.791476 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.791442    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"]
Sep 19 08:26:53.807112 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.807079    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"]
Sep 19 08:26:53.807347 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.807218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:53.809852 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.809830    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 19 08:26:53.831150 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.831124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m984t\" (UniqueName: \"kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-kube-api-access-m984t\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:53.831276 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.831212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-certificates\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:53.831276 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.831247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-cabundle0\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:53.831396 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:53.831377    2577 secret.go:281] references non-existent secret key: ca.crt
Sep 19 08:26:53.831455 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:53.831401    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 19 08:26:53.831455 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:53.831412    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5sgdl: references non-existent secret key: ca.crt
Sep 19 08:26:53.831553 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:53.831469    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-certificates podName:cb7f7ff1-6be0-4a92-be4f-2607d34f45fa nodeName:}" failed. No retries permitted until 2026-09-19 08:26:54.331451324 +0000 UTC m=+382.013034746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-certificates") pod "keda-operator-579c8958f7-5sgdl" (UID: "cb7f7ff1-6be0-4a92-be4f-2607d34f45fa") : references non-existent secret key: ca.crt
Sep 19 08:26:53.831865 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.831847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-cabundle0\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:53.838853 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.838802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m984t\" (UniqueName: \"kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-kube-api-access-m984t\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:53.932111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.932075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvks\" (UniqueName: \"kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-kube-api-access-6zvks\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:53.932274 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.932137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:53.932274 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:53.932191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/52c771ad-90b6-4602-adb4-955e8c610e0d-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:54.033615 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.033583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvks\" (UniqueName: \"kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-kube-api-access-6zvks\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:54.033984 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.033642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:54.033984 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.033673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/52c771ad-90b6-4602-adb4-955e8c610e0d-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:54.033984 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.033789    2577 secret.go:281] references non-existent secret key: tls.crt
Sep 19 08:26:54.033984 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.033809    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 19 08:26:54.033984 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.033829    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q: references non-existent secret key: tls.crt
Sep 19 08:26:54.033984 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.033884    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-certificates podName:52c771ad-90b6-4602-adb4-955e8c610e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:26:54.533869018 +0000 UTC m=+382.215452451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-certificates") pod "keda-metrics-apiserver-77857b7b98-9bd2q" (UID: "52c771ad-90b6-4602-adb4-955e8c610e0d") : references non-existent secret key: tls.crt
Sep 19 08:26:54.034172 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.034099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/52c771ad-90b6-4602-adb4-955e8c610e0d-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:54.042356 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.042329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvks\" (UniqueName: \"kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-kube-api-access-6zvks\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:54.336083 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.336055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-certificates\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:54.336256 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.336211    2577 secret.go:281] references non-existent secret key: ca.crt
Sep 19 08:26:54.336256 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.336234    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 19 08:26:54.336256 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.336244    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5sgdl: references non-existent secret key: ca.crt
Sep 19 08:26:54.336354 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.336296    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-certificates podName:cb7f7ff1-6be0-4a92-be4f-2607d34f45fa nodeName:}" failed. No retries permitted until 2026-09-19 08:26:55.336281667 +0000 UTC m=+383.017865078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-certificates") pod "keda-operator-579c8958f7-5sgdl" (UID: "cb7f7ff1-6be0-4a92-be4f-2607d34f45fa") : references non-existent secret key: ca.crt
Sep 19 08:26:54.538412 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.538378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:54.538567 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.538545    2577 secret.go:281] references non-existent secret key: tls.crt
Sep 19 08:26:54.538607 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.538569    2577 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 19 08:26:54.538607 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.538592    2577 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q: references non-existent secret key: tls.crt
Sep 19 08:26:54.538666 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:26:54.538655    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-certificates podName:52c771ad-90b6-4602-adb4-955e8c610e0d nodeName:}" failed. No retries permitted until 2026-09-19 08:26:55.538635665 +0000 UTC m=+383.220219093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-certificates") pod "keda-metrics-apiserver-77857b7b98-9bd2q" (UID: "52c771ad-90b6-4602-adb4-955e8c610e0d") : references non-existent secret key: tls.crt
Sep 19 08:26:54.660325 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.660295    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-r88m9"]
Sep 19 08:26:54.685035 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.685013    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-r88m9"]
Sep 19 08:26:54.685180 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.685101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:26:54.687532 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.687508    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 19 08:26:54.840354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.840320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d219426c-a2f1-45a9-a97e-b1e9a94344a7-certificates\") pod \"keda-admission-6c4d864b86-r88m9\" (UID: \"d219426c-a2f1-45a9-a97e-b1e9a94344a7\") " pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:26:54.840545 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.840430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8m4\" (UniqueName: \"kubernetes.io/projected/d219426c-a2f1-45a9-a97e-b1e9a94344a7-kube-api-access-5j8m4\") pod \"keda-admission-6c4d864b86-r88m9\" (UID: \"d219426c-a2f1-45a9-a97e-b1e9a94344a7\") " pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:26:54.941323 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.941227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8m4\" (UniqueName: \"kubernetes.io/projected/d219426c-a2f1-45a9-a97e-b1e9a94344a7-kube-api-access-5j8m4\") pod \"keda-admission-6c4d864b86-r88m9\" (UID: \"d219426c-a2f1-45a9-a97e-b1e9a94344a7\") " pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:26:54.941497 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.941409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d219426c-a2f1-45a9-a97e-b1e9a94344a7-certificates\") pod \"keda-admission-6c4d864b86-r88m9\" (UID: \"d219426c-a2f1-45a9-a97e-b1e9a94344a7\") " pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:26:54.944005 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.943972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d219426c-a2f1-45a9-a97e-b1e9a94344a7-certificates\") pod \"keda-admission-6c4d864b86-r88m9\" (UID: \"d219426c-a2f1-45a9-a97e-b1e9a94344a7\") " pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:26:54.950156 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.950134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8m4\" (UniqueName: \"kubernetes.io/projected/d219426c-a2f1-45a9-a97e-b1e9a94344a7-kube-api-access-5j8m4\") pod \"keda-admission-6c4d864b86-r88m9\" (UID: \"d219426c-a2f1-45a9-a97e-b1e9a94344a7\") " pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:26:54.995104 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:54.995069    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:26:55.273339 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:55.273311    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-r88m9"]
Sep 19 08:26:55.276198 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:26:55.276169    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd219426c_a2f1_45a9_a97e_b1e9a94344a7.slice/crio-dc4003fb2758899b676204116cbbb558ae1c44832ce8d259208a67c300293d77 WatchSource:0}: Error finding container dc4003fb2758899b676204116cbbb558ae1c44832ce8d259208a67c300293d77: Status 404 returned error can't find the container with id dc4003fb2758899b676204116cbbb558ae1c44832ce8d259208a67c300293d77
Sep 19 08:26:55.345647 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:55.345612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-certificates\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:55.348059 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:55.348020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/cb7f7ff1-6be0-4a92-be4f-2607d34f45fa-certificates\") pod \"keda-operator-579c8958f7-5sgdl\" (UID: \"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa\") " pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:55.420834 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:55.420804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:26:55.547185 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:55.547095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:55.549645 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:55.549617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/52c771ad-90b6-4602-adb4-955e8c610e0d-certificates\") pod \"keda-metrics-apiserver-77857b7b98-9bd2q\" (UID: \"52c771ad-90b6-4602-adb4-955e8c610e0d\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:55.620541 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:55.620505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:26:55.692155 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:55.692128    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5sgdl"]
Sep 19 08:26:55.695291 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:26:55.695257    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7f7ff1_6be0_4a92_be4f_2607d34f45fa.slice/crio-8bb4b0773de5fde2445d5356d8a7c0a9734c9c72d719d87363f724d543cbddfc WatchSource:0}: Error finding container 8bb4b0773de5fde2445d5356d8a7c0a9734c9c72d719d87363f724d543cbddfc: Status 404 returned error can't find the container with id 8bb4b0773de5fde2445d5356d8a7c0a9734c9c72d719d87363f724d543cbddfc
Sep 19 08:26:55.892573 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:55.892544    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"]
Sep 19 08:26:55.894544 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:26:55.894514    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c771ad_90b6_4602_adb4_955e8c610e0d.slice/crio-35c1c055c62ab4d3853c4ccd2a32b05552652d96b2558ab9f1a656f612c8b6e9 WatchSource:0}: Error finding container 35c1c055c62ab4d3853c4ccd2a32b05552652d96b2558ab9f1a656f612c8b6e9: Status 404 returned error can't find the container with id 35c1c055c62ab4d3853c4ccd2a32b05552652d96b2558ab9f1a656f612c8b6e9
Sep 19 08:26:56.015713 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:56.015676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-r88m9" event={"ID":"d219426c-a2f1-45a9-a97e-b1e9a94344a7","Type":"ContainerStarted","Data":"dc4003fb2758899b676204116cbbb558ae1c44832ce8d259208a67c300293d77"}
Sep 19 08:26:56.017180 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:56.017152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q" event={"ID":"52c771ad-90b6-4602-adb4-955e8c610e0d","Type":"ContainerStarted","Data":"35c1c055c62ab4d3853c4ccd2a32b05552652d96b2558ab9f1a656f612c8b6e9"}
Sep 19 08:26:56.018533 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:56.018504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5sgdl" event={"ID":"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa","Type":"ContainerStarted","Data":"8bb4b0773de5fde2445d5356d8a7c0a9734c9c72d719d87363f724d543cbddfc"}
Sep 19 08:26:57.026888 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:57.026851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-r88m9" event={"ID":"d219426c-a2f1-45a9-a97e-b1e9a94344a7","Type":"ContainerStarted","Data":"2160f5f46f9bc89d3276c66cfd91f01835b3ba4d5a05bb56122ac4e1346a115c"}
Sep 19 08:26:57.027397 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:57.026977    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:26:57.044704 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:26:57.044640    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-r88m9" podStartSLOduration=1.8284189290000001 podStartE2EDuration="3.04461799s" totalImagesPullingTime="1.216199061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:57.043119609 +0000 UTC m=+384.724703046" watchObservedRunningTime="2026-09-19 08:26:57.04461799 +0000 UTC m=+384.726201423"
Sep 19 08:27:00.046911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:00.046881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q" event={"ID":"52c771ad-90b6-4602-adb4-955e8c610e0d","Type":"ContainerStarted","Data":"2ac507cc4a82ec74d863f86750dde0b6f4d8636e7223063fde5437225b5f833e"}
Sep 19 08:27:00.047575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:00.047047    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:27:00.048526 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:00.048501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5sgdl" event={"ID":"cb7f7ff1-6be0-4a92-be4f-2607d34f45fa","Type":"ContainerStarted","Data":"be0cf046cdbcfa54d28f40f1085f3b27c6b9b6b2cc1fa039c30a08545cac4211"}
Sep 19 08:27:00.048643 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:00.048543    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:27:00.065049 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:00.065008    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q" podStartSLOduration=3.4088326159999998 podStartE2EDuration="7.06499372s" totalImagesPullingTime="3.656161104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:00.06438117 +0000 UTC m=+387.745964596" watchObservedRunningTime="2026-09-19 08:27:00.06499372 +0000 UTC m=+387.746577153"
Sep 19 08:27:11.058509 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:11.058474    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-9bd2q"
Sep 19 08:27:11.095673 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:11.095618    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-5sgdl" podStartSLOduration=14.234201027 podStartE2EDuration="18.095602927s" totalImagesPullingTime="3.8614019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:00.0836823 +0000 UTC m=+387.765265733" watchObservedRunningTime="2026-09-19 08:27:11.095602927 +0000 UTC m=+398.777186372"
Sep 19 08:27:14.007470 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:14.007441    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-pqv7d"
Sep 19 08:27:18.034197 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:18.034168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-r88m9"
Sep 19 08:27:21.056832 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:21.056806    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-5sgdl"
Sep 19 08:27:46.281441 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.281352    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"]
Sep 19 08:27:46.292004 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.291980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.294206 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.294177    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"]
Sep 19 08:27:46.294954 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.294925    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 19 08:27:46.295083 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.294952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 19 08:27:46.295083 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.294960    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7lq9v\""
Sep 19 08:27:46.457690 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.457660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.457851 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.457697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.457851 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.457746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrxn\" (UniqueName: \"kubernetes.io/projected/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-kube-api-access-rgrxn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.559013 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.558940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrxn\" (UniqueName: \"kubernetes.io/projected/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-kube-api-access-rgrxn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.559013 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.558991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.559177 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.559017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.559311 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.559297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.559413 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.559396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.566934 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.566905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrxn\" (UniqueName: \"kubernetes.io/projected/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-kube-api-access-rgrxn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.602717 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.602694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:46.874863 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:46.874833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"]
Sep 19 08:27:46.878144 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:27:46.878116    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cddcce8_b343_4a75_bc70_723c1c2b6ce6.slice/crio-77ebeeed85055e52c7038fc527b5b0d4ad722c6ac4c7d733e730fa0c061a878b WatchSource:0}: Error finding container 77ebeeed85055e52c7038fc527b5b0d4ad722c6ac4c7d733e730fa0c061a878b: Status 404 returned error can't find the container with id 77ebeeed85055e52c7038fc527b5b0d4ad722c6ac4c7d733e730fa0c061a878b
Sep 19 08:27:47.257099 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:47.257009    2577 generic.go:356] "Generic (PLEG): container finished" podID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerID="14ff462c48824aa22ece8af027743f57a59b45725ea722eaa9d5ab178d0e2bab" exitCode=0
Sep 19 08:27:47.257424 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:47.257095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw" event={"ID":"2cddcce8-b343-4a75-bc70-723c1c2b6ce6","Type":"ContainerDied","Data":"14ff462c48824aa22ece8af027743f57a59b45725ea722eaa9d5ab178d0e2bab"}
Sep 19 08:27:47.257424 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:47.257143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw" event={"ID":"2cddcce8-b343-4a75-bc70-723c1c2b6ce6","Type":"ContainerStarted","Data":"77ebeeed85055e52c7038fc527b5b0d4ad722c6ac4c7d733e730fa0c061a878b"}
Sep 19 08:27:50.272231 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:50.272191    2577 generic.go:356] "Generic (PLEG): container finished" podID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerID="2ecce2e29825c73488d94d1d68dacd9477c294d32d73aa571a1e17277faa4ab8" exitCode=0
Sep 19 08:27:50.272627 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:50.272256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw" event={"ID":"2cddcce8-b343-4a75-bc70-723c1c2b6ce6","Type":"ContainerDied","Data":"2ecce2e29825c73488d94d1d68dacd9477c294d32d73aa571a1e17277faa4ab8"}
Sep 19 08:27:51.278980 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:51.278944    2577 generic.go:356] "Generic (PLEG): container finished" podID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerID="8c611cf72bb6aaad64434d775a3ad508b4aa5bfdccd6df2a6a9fab6eb7909fb3" exitCode=0
Sep 19 08:27:51.279423 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:51.279028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw" event={"ID":"2cddcce8-b343-4a75-bc70-723c1c2b6ce6","Type":"ContainerDied","Data":"8c611cf72bb6aaad64434d775a3ad508b4aa5bfdccd6df2a6a9fab6eb7909fb3"}
Sep 19 08:27:52.403521 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.403495    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:52.511288 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.511263    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-bundle\") pod \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") "
Sep 19 08:27:52.511436 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.511323    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-util\") pod \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") "
Sep 19 08:27:52.511436 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.511393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgrxn\" (UniqueName: \"kubernetes.io/projected/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-kube-api-access-rgrxn\") pod \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\" (UID: \"2cddcce8-b343-4a75-bc70-723c1c2b6ce6\") "
Sep 19 08:27:52.512031 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.512001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-bundle" (OuterVolumeSpecName: "bundle") pod "2cddcce8-b343-4a75-bc70-723c1c2b6ce6" (UID: "2cddcce8-b343-4a75-bc70-723c1c2b6ce6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:27:52.513464 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.513441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-kube-api-access-rgrxn" (OuterVolumeSpecName: "kube-api-access-rgrxn") pod "2cddcce8-b343-4a75-bc70-723c1c2b6ce6" (UID: "2cddcce8-b343-4a75-bc70-723c1c2b6ce6"). InnerVolumeSpecName "kube-api-access-rgrxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:27:52.540967 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.540913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-util" (OuterVolumeSpecName: "util") pod "2cddcce8-b343-4a75-bc70-723c1c2b6ce6" (UID: "2cddcce8-b343-4a75-bc70-723c1c2b6ce6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:27:52.612564 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.612540    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-util\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:27:52.612564 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.612561    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgrxn\" (UniqueName: \"kubernetes.io/projected/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-kube-api-access-rgrxn\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:27:52.612691 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:52.612572    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cddcce8-b343-4a75-bc70-723c1c2b6ce6-bundle\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:27:53.288310 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:53.288282    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw"
Sep 19 08:27:53.288498 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:53.288282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540msgsw" event={"ID":"2cddcce8-b343-4a75-bc70-723c1c2b6ce6","Type":"ContainerDied","Data":"77ebeeed85055e52c7038fc527b5b0d4ad722c6ac4c7d733e730fa0c061a878b"}
Sep 19 08:27:53.288498 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:53.288411    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77ebeeed85055e52c7038fc527b5b0d4ad722c6ac4c7d733e730fa0c061a878b"
Sep 19 08:27:59.280010 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.279971    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"]
Sep 19 08:27:59.280483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.280322    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerName="util"
Sep 19 08:27:59.280483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.280338    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerName="util"
Sep 19 08:27:59.280483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.280350    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerName="extract"
Sep 19 08:27:59.280483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.280358    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerName="extract"
Sep 19 08:27:59.280483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.280399    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerName="pull"
Sep 19 08:27:59.280483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.280407    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerName="pull"
Sep 19 08:27:59.280483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.280473    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cddcce8-b343-4a75-bc70-723c1c2b6ce6" containerName="extract"
Sep 19 08:27:59.284487 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.284468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"
Sep 19 08:27:59.287837 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.287817    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 19 08:27:59.288268 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.288252    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jt44p\""
Sep 19 08:27:59.288510 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.288492    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:27:59.299351 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.299329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"]
Sep 19 08:27:59.366275 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.366241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/159c23e9-a334-4f47-8b97-67c13fe225cf-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jxrcc\" (UID: \"159c23e9-a334-4f47-8b97-67c13fe225cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"
Sep 19 08:27:59.366275 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.366274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28ft\" (UniqueName: \"kubernetes.io/projected/159c23e9-a334-4f47-8b97-67c13fe225cf-kube-api-access-c28ft\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jxrcc\" (UID: \"159c23e9-a334-4f47-8b97-67c13fe225cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"
Sep 19 08:27:59.466699 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.466667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/159c23e9-a334-4f47-8b97-67c13fe225cf-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jxrcc\" (UID: \"159c23e9-a334-4f47-8b97-67c13fe225cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"
Sep 19 08:27:59.466854 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.466704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28ft\" (UniqueName: \"kubernetes.io/projected/159c23e9-a334-4f47-8b97-67c13fe225cf-kube-api-access-c28ft\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jxrcc\" (UID: \"159c23e9-a334-4f47-8b97-67c13fe225cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"
Sep 19 08:27:59.467114 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.467094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/159c23e9-a334-4f47-8b97-67c13fe225cf-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jxrcc\" (UID: \"159c23e9-a334-4f47-8b97-67c13fe225cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"
Sep 19 08:27:59.475821 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.475797    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28ft\" (UniqueName: \"kubernetes.io/projected/159c23e9-a334-4f47-8b97-67c13fe225cf-kube-api-access-c28ft\") pod \"cert-manager-operator-controller-manager-684f79c9b9-jxrcc\" (UID: \"159c23e9-a334-4f47-8b97-67c13fe225cf\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"
Sep 19 08:27:59.593705 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.593621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"
Sep 19 08:27:59.889624 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:27:59.888406    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc"]
Sep 19 08:27:59.893505 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:27:59.893447    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159c23e9_a334_4f47_8b97_67c13fe225cf.slice/crio-b8004d598c807c406ec71092ea4f611cf98ff55a86b6bea411cdf5a4c5d8acb8 WatchSource:0}: Error finding container b8004d598c807c406ec71092ea4f611cf98ff55a86b6bea411cdf5a4c5d8acb8: Status 404 returned error can't find the container with id b8004d598c807c406ec71092ea4f611cf98ff55a86b6bea411cdf5a4c5d8acb8
Sep 19 08:28:00.318439 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:00.318331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc" event={"ID":"159c23e9-a334-4f47-8b97-67c13fe225cf","Type":"ContainerStarted","Data":"b8004d598c807c406ec71092ea4f611cf98ff55a86b6bea411cdf5a4c5d8acb8"}
Sep 19 08:28:02.332746 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:02.332701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc" event={"ID":"159c23e9-a334-4f47-8b97-67c13fe225cf","Type":"ContainerStarted","Data":"fe0fc19f0530de905105dee02bb22b6f3853588fc40fe72c0fad220b9d550752"}
Sep 19 08:28:02.361969 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:02.361909    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-jxrcc" podStartSLOduration=1.4172968830000001 podStartE2EDuration="3.361893088s" totalImagesPullingTime="1.944596205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:28:02.361763249 +0000 UTC m=+450.043346683" watchObservedRunningTime="2026-09-19 08:28:02.361893088 +0000 UTC m=+450.043476521"
Sep 19 08:28:08.333853 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.333815    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"]
Sep 19 08:28:08.337444 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.337424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.340008 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.339983    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 19 08:28:08.340113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.339986    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7lq9v\""
Sep 19 08:28:08.340872 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.340858    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 19 08:28:08.345496 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.345474    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"]
Sep 19 08:28:08.431773 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.431740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpqv\" (UniqueName: \"kubernetes.io/projected/f7306463-4af2-4936-a1f4-275b95afcb5b-kube-api-access-btpqv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.431773 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.431779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.431982 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.431808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.532475 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.532437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpqv\" (UniqueName: \"kubernetes.io/projected/f7306463-4af2-4936-a1f4-275b95afcb5b-kube-api-access-btpqv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.532646 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.532515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.532646 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.532583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.532957 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.532937    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.533033 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.532970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.547428 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.547398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpqv\" (UniqueName: \"kubernetes.io/projected/f7306463-4af2-4936-a1f4-275b95afcb5b-kube-api-access-btpqv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.647578 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.647543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:08.949916 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:08.949893    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"]
Sep 19 08:28:08.951874 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:28:08.951845    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7306463_4af2_4936_a1f4_275b95afcb5b.slice/crio-67a29939003ad8e42056e564bf54f681e09c4a67f486f717a4a0cdc9ecaa8ef9 WatchSource:0}: Error finding container 67a29939003ad8e42056e564bf54f681e09c4a67f486f717a4a0cdc9ecaa8ef9: Status 404 returned error can't find the container with id 67a29939003ad8e42056e564bf54f681e09c4a67f486f717a4a0cdc9ecaa8ef9
Sep 19 08:28:09.366631 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:09.366596    2577 generic.go:356] "Generic (PLEG): container finished" podID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerID="dc8be6317d1c2016721d289b643112688943b1788cddbf921a1a9937eeb2fd2f" exitCode=0
Sep 19 08:28:09.366986 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:09.366643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv" event={"ID":"f7306463-4af2-4936-a1f4-275b95afcb5b","Type":"ContainerDied","Data":"dc8be6317d1c2016721d289b643112688943b1788cddbf921a1a9937eeb2fd2f"}
Sep 19 08:28:09.366986 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:09.366689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv" event={"ID":"f7306463-4af2-4936-a1f4-275b95afcb5b","Type":"ContainerStarted","Data":"67a29939003ad8e42056e564bf54f681e09c4a67f486f717a4a0cdc9ecaa8ef9"}
Sep 19 08:28:11.382608 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:11.382579    2577 generic.go:356] "Generic (PLEG): container finished" podID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerID="d02568def4653df45317b0d8569c06c0a9798a627f0b5f0a270b1d86ce2c41e6" exitCode=0
Sep 19 08:28:11.382916 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:11.382657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv" event={"ID":"f7306463-4af2-4936-a1f4-275b95afcb5b","Type":"ContainerDied","Data":"d02568def4653df45317b0d8569c06c0a9798a627f0b5f0a270b1d86ce2c41e6"}
Sep 19 08:28:12.389075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:12.389039    2577 generic.go:356] "Generic (PLEG): container finished" podID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerID="27864577e9b24e8f524dfbddf31499389cd76275e3533ece84985d53beaa3dcf" exitCode=0
Sep 19 08:28:12.389468 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:12.389123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv" event={"ID":"f7306463-4af2-4936-a1f4-275b95afcb5b","Type":"ContainerDied","Data":"27864577e9b24e8f524dfbddf31499389cd76275e3533ece84985d53beaa3dcf"}
Sep 19 08:28:13.517573 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.517550    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:13.579183 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.579156    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-bundle\") pod \"f7306463-4af2-4936-a1f4-275b95afcb5b\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") "
Sep 19 08:28:13.579373 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.579214    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-util\") pod \"f7306463-4af2-4936-a1f4-275b95afcb5b\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") "
Sep 19 08:28:13.579373 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.579246    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btpqv\" (UniqueName: \"kubernetes.io/projected/f7306463-4af2-4936-a1f4-275b95afcb5b-kube-api-access-btpqv\") pod \"f7306463-4af2-4936-a1f4-275b95afcb5b\" (UID: \"f7306463-4af2-4936-a1f4-275b95afcb5b\") "
Sep 19 08:28:13.579635 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.579609    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-bundle" (OuterVolumeSpecName: "bundle") pod "f7306463-4af2-4936-a1f4-275b95afcb5b" (UID: "f7306463-4af2-4936-a1f4-275b95afcb5b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:13.581921 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.581891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7306463-4af2-4936-a1f4-275b95afcb5b-kube-api-access-btpqv" (OuterVolumeSpecName: "kube-api-access-btpqv") pod "f7306463-4af2-4936-a1f4-275b95afcb5b" (UID: "f7306463-4af2-4936-a1f4-275b95afcb5b"). InnerVolumeSpecName "kube-api-access-btpqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:28:13.586872 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.586841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-util" (OuterVolumeSpecName: "util") pod "f7306463-4af2-4936-a1f4-275b95afcb5b" (UID: "f7306463-4af2-4936-a1f4-275b95afcb5b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:13.680505 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.680419    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-bundle\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:13.680505 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.680449    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7306463-4af2-4936-a1f4-275b95afcb5b-util\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:13.680505 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:13.680458    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpqv\" (UniqueName: \"kubernetes.io/projected/f7306463-4af2-4936-a1f4-275b95afcb5b-kube-api-access-btpqv\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:14.399041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:14.399009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv" event={"ID":"f7306463-4af2-4936-a1f4-275b95afcb5b","Type":"ContainerDied","Data":"67a29939003ad8e42056e564bf54f681e09c4a67f486f717a4a0cdc9ecaa8ef9"}
Sep 19 08:28:14.399210 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:14.399054    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a29939003ad8e42056e564bf54f681e09c4a67f486f717a4a0cdc9ecaa8ef9"
Sep 19 08:28:14.399210 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:14.399025    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcc9wv"
Sep 19 08:28:35.764228 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.764196    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"]
Sep 19 08:28:35.764684 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.764507    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerName="extract"
Sep 19 08:28:35.764684 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.764518    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerName="extract"
Sep 19 08:28:35.764684 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.764528    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerName="pull"
Sep 19 08:28:35.764684 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.764534    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerName="pull"
Sep 19 08:28:35.764684 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.764544    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerName="util"
Sep 19 08:28:35.764684 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.764549    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerName="util"
Sep 19 08:28:35.764684 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.764607    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7306463-4af2-4936-a1f4-275b95afcb5b" containerName="extract"
Sep 19 08:28:35.772988 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.772960    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.775614 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.775589    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 19 08:28:35.775614 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.775603    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cqs2t\""
Sep 19 08:28:35.775800 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.775588    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 19 08:28:35.775800 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.775630    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 19 08:28:35.775800 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.775600    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 19 08:28:35.783290 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.783266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"]
Sep 19 08:28:35.866772 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.866726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/137e70de-a464-443d-b131-36d4c901c033-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.867003 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.866835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.867003 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.866896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.867003 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.866921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/137e70de-a464-443d-b131-36d4c901c033-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.867226 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.867029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.867226 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.867074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.867226 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.867113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxzwz\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-kube-api-access-zxzwz\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.967894 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.967856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/137e70de-a464-443d-b131-36d4c901c033-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.968070 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.967901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.968070 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.967932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.968070 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.967984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/137e70de-a464-443d-b131-36d4c901c033-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.968070 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.968044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.968275 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.968071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.968275 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.968097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxzwz\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-kube-api-access-zxzwz\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.968726 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.968696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/137e70de-a464-443d-b131-36d4c901c033-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.970179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.970158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/137e70de-a464-443d-b131-36d4c901c033-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.970309 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.970293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.970643 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.970627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.970777 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.970760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.975805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.975777    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxzwz\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-kube-api-access-zxzwz\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:35.975902 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:35.975881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kvm9w\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:36.084105 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:36.083997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:36.360491 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:36.360425    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"]
Sep 19 08:28:36.362511 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:28:36.362470    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137e70de_a464_443d_b131_36d4c901c033.slice/crio-e48e63858b266f72552f200fea9cd102ae3d77b85c22a112ea1ca1a2d31defec WatchSource:0}: Error finding container e48e63858b266f72552f200fea9cd102ae3d77b85c22a112ea1ca1a2d31defec: Status 404 returned error can't find the container with id e48e63858b266f72552f200fea9cd102ae3d77b85c22a112ea1ca1a2d31defec
Sep 19 08:28:36.497262 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:36.497228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w" event={"ID":"137e70de-a464-443d-b131-36d4c901c033","Type":"ContainerStarted","Data":"e48e63858b266f72552f200fea9cd102ae3d77b85c22a112ea1ca1a2d31defec"}
Sep 19 08:28:38.887343 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:38.887301    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:28:38.887645 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:38.887383    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:28:39.515713 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:39.515677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w" event={"ID":"137e70de-a464-443d-b131-36d4c901c033","Type":"ContainerStarted","Data":"b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397"}
Sep 19 08:28:39.515911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:39.515846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:39.517759 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:39.517729    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-kvm9w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 19 08:28:39.517882 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:39.517788    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w" podUID="137e70de-a464-443d-b131-36d4c901c033" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:28:39.536129 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:39.536075    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w" podStartSLOduration=2.013570585 podStartE2EDuration="4.53605976s" totalImagesPullingTime="2.522489175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:28:39.534353702 +0000 UTC m=+487.215937161" watchObservedRunningTime="2026-09-19 08:28:39.53605976 +0000 UTC m=+487.217643192"
Sep 19 08:28:40.520170 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:40.520138    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:28:42.325249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.325213    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"]
Sep 19 08:28:42.328933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.328905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.331477 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.331440    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-j5dfl\""
Sep 19 08:28:42.341995 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.341963    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"]
Sep 19 08:28:42.424305 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.424272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxdq\" (UniqueName: \"kubernetes.io/projected/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-kube-api-access-cbxdq\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.424497 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.424315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.424497 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.424345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.424497 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.424392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.424497 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.424420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.424497 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.424440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.424685 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.424519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.424685 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.424567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.424685 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.424602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525438 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525659    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxdq\" (UniqueName: \"kubernetes.io/projected/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-kube-api-access-cbxdq\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.525962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.525866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.526104 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.526073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.526161 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.526096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.526161 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.526125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.526305 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.526280    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.528964 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.528633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.528964 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.528879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.535547 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.535521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.538105 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.538072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxdq\" (UniqueName: \"kubernetes.io/projected/f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67-kube-api-access-cbxdq\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl\" (UID: \"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.643639 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.643603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:42.922029 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:42.922005    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"]
Sep 19 08:28:42.924329 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:28:42.924301    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b1ce6b_ad39_472b_bdc5_ee4bb2b76a67.slice/crio-7bb54b7cce36739560bac9c7e012fb11ad3b119d77a522d5206ed8f405e8eeb6 WatchSource:0}: Error finding container 7bb54b7cce36739560bac9c7e012fb11ad3b119d77a522d5206ed8f405e8eeb6: Status 404 returned error can't find the container with id 7bb54b7cce36739560bac9c7e012fb11ad3b119d77a522d5206ed8f405e8eeb6
Sep 19 08:28:43.536895 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:43.536860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" event={"ID":"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67","Type":"ContainerStarted","Data":"7bb54b7cce36739560bac9c7e012fb11ad3b119d77a522d5206ed8f405e8eeb6"}
Sep 19 08:28:45.736188 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:45.736151    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:28:45.736519 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:45.736221    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:28:45.736519 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:45.736260    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:28:46.554655 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:46.554618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" event={"ID":"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67","Type":"ContainerStarted","Data":"51e7e6b0a015ca01432c5a67b51bb186ab5c1e8b3109943b3e679f793c414c5a"}
Sep 19 08:28:46.644313 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:46.644278    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:46.649238 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:46.649213    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:46.678431 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:46.673795    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" podStartSLOduration=1.864495434 podStartE2EDuration="4.673774581s" totalImagesPullingTime="2.809279147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:28:46.608217598 +0000 UTC m=+494.289801042" watchObservedRunningTime="2026-09-19 08:28:46.673774581 +0000 UTC m=+494.355358016"
Sep 19 08:28:47.559353 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:47.559318    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:47.560331 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:47.560310    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:28:51.411156 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.411123    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"]
Sep 19 08:28:51.414600 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.414582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.417428 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.417410    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 19 08:28:51.417543 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.417445    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7lq9v\""
Sep 19 08:28:51.417543 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.417463    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 19 08:28:51.424509 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.424487    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"]
Sep 19 08:28:51.499127 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.499094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.499321 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.499161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxsh\" (UniqueName: \"kubernetes.io/projected/be456d74-a1e7-485d-a144-ddc9436cf2ff-kube-api-access-jrxsh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.499321 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.499196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.512235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.512202    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"]
Sep 19 08:28:51.515654 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.515638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.524566 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.524546    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"]
Sep 19 08:28:51.600189 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.600164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mrd\" (UniqueName: \"kubernetes.io/projected/ba804a76-78b8-4598-a53e-ec7f529bb562-kube-api-access-86mrd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.600317 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.600197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxsh\" (UniqueName: \"kubernetes.io/projected/be456d74-a1e7-485d-a144-ddc9436cf2ff-kube-api-access-jrxsh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.600317 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.600230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.600317 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.600276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.600317 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.600300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.600499 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.600425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.600589 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.600574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.600711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.600695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.608763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.608739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxsh\" (UniqueName: \"kubernetes.io/projected/be456d74-a1e7-485d-a144-ddc9436cf2ff-kube-api-access-jrxsh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.612615 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.612592    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"]
Sep 19 08:28:51.616111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.616095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.625460 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.625437    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"]
Sep 19 08:28:51.701075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.700981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mrd\" (UniqueName: \"kubernetes.io/projected/ba804a76-78b8-4598-a53e-ec7f529bb562-kube-api-access-86mrd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.701075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.701044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.701075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.701067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.701313 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.701109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.701313 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.701136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpdm\" (UniqueName: \"kubernetes.io/projected/24e9fda7-4baa-4709-80d3-e40c780998f6-kube-api-access-4gpdm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.701313 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.701171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.701538 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.701517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.701575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.701527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.712013 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.711989    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"]
Sep 19 08:28:51.715442 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.715425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.716638 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.716619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mrd\" (UniqueName: \"kubernetes.io/projected/ba804a76-78b8-4598-a53e-ec7f529bb562-kube-api-access-86mrd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.723881 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.723864    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:51.730198 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.730164    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"]
Sep 19 08:28:51.802082 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.802053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sg2q\" (UniqueName: \"kubernetes.io/projected/3a6bc187-4a2e-4520-939f-d39538bbd9d9-kube-api-access-6sg2q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.802222 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.802135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.802222 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.802162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.802222 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.802199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpdm\" (UniqueName: \"kubernetes.io/projected/24e9fda7-4baa-4709-80d3-e40c780998f6-kube-api-access-4gpdm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.802409 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.802237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.802409 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.802278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.802539 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.802518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.802587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.802540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.811056 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.811035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpdm\" (UniqueName: \"kubernetes.io/projected/24e9fda7-4baa-4709-80d3-e40c780998f6-kube-api-access-4gpdm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.824888 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.824865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:51.902977 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.902943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.903087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.903039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sg2q\" (UniqueName: \"kubernetes.io/projected/3a6bc187-4a2e-4520-939f-d39538bbd9d9-kube-api-access-6sg2q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.903135 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.903118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.903359 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.903336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.903465 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.903442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.919553 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.919528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sg2q\" (UniqueName: \"kubernetes.io/projected/3a6bc187-4a2e-4520-939f-d39538bbd9d9-kube-api-access-6sg2q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:51.926609 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.926557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:51.999544 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:51.999513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"]
Sep 19 08:28:52.002583 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:28:52.002551    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe456d74_a1e7_485d_a144_ddc9436cf2ff.slice/crio-1a974075663288cbbcbd3f9766157e11e8747915cb0eeed1aed36aad7c3fbccd WatchSource:0}: Error finding container 1a974075663288cbbcbd3f9766157e11e8747915cb0eeed1aed36aad7c3fbccd: Status 404 returned error can't find the container with id 1a974075663288cbbcbd3f9766157e11e8747915cb0eeed1aed36aad7c3fbccd
Sep 19 08:28:52.032952 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.032926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:52.098381 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.098342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"]
Sep 19 08:28:52.101078 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:28:52.101045    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba804a76_78b8_4598_a53e_ec7f529bb562.slice/crio-93302586d2c46a8cc4e43e27dc4d4b276cd49b1c89b7dd34c8d09a41d380693f WatchSource:0}: Error finding container 93302586d2c46a8cc4e43e27dc4d4b276cd49b1c89b7dd34c8d09a41d380693f: Status 404 returned error can't find the container with id 93302586d2c46a8cc4e43e27dc4d4b276cd49b1c89b7dd34c8d09a41d380693f
Sep 19 08:28:52.196990 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.196962    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"]
Sep 19 08:28:52.198568 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:28:52.198542    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e9fda7_4baa_4709_80d3_e40c780998f6.slice/crio-f09f62d1920446bd576d43a06012f67508d9114f9e4f399def5d49ff4faef81d WatchSource:0}: Error finding container f09f62d1920446bd576d43a06012f67508d9114f9e4f399def5d49ff4faef81d: Status 404 returned error can't find the container with id f09f62d1920446bd576d43a06012f67508d9114f9e4f399def5d49ff4faef81d
Sep 19 08:28:52.302740 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.302714    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"]
Sep 19 08:28:52.378210 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:28:52.378177    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6bc187_4a2e_4520_939f_d39538bbd9d9.slice/crio-0e843d86e61b6660b621f2442aebcd54425eeed1b7ef6970c1c78a94493c6e8d WatchSource:0}: Error finding container 0e843d86e61b6660b621f2442aebcd54425eeed1b7ef6970c1c78a94493c6e8d: Status 404 returned error can't find the container with id 0e843d86e61b6660b621f2442aebcd54425eeed1b7ef6970c1c78a94493c6e8d
Sep 19 08:28:52.583297 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.583214    2577 generic.go:356] "Generic (PLEG): container finished" podID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerID="fd7a3426da91cf9f45573b7177104f127740a6f7fad7875dcdf93f8d16b21d9b" exitCode=0
Sep 19 08:28:52.583706 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.583303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2" event={"ID":"3a6bc187-4a2e-4520-939f-d39538bbd9d9","Type":"ContainerDied","Data":"fd7a3426da91cf9f45573b7177104f127740a6f7fad7875dcdf93f8d16b21d9b"}
Sep 19 08:28:52.583706 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.583352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2" event={"ID":"3a6bc187-4a2e-4520-939f-d39538bbd9d9","Type":"ContainerStarted","Data":"0e843d86e61b6660b621f2442aebcd54425eeed1b7ef6970c1c78a94493c6e8d"}
Sep 19 08:28:52.584931 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.584913    2577 generic.go:356] "Generic (PLEG): container finished" podID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerID="c0f8f593ed55161fb64f44ce3a59884b247b1503d68a137d9fc24c41e2d04903" exitCode=0
Sep 19 08:28:52.585021 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.584939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4" event={"ID":"ba804a76-78b8-4598-a53e-ec7f529bb562","Type":"ContainerDied","Data":"c0f8f593ed55161fb64f44ce3a59884b247b1503d68a137d9fc24c41e2d04903"}
Sep 19 08:28:52.585021 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.584976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4" event={"ID":"ba804a76-78b8-4598-a53e-ec7f529bb562","Type":"ContainerStarted","Data":"93302586d2c46a8cc4e43e27dc4d4b276cd49b1c89b7dd34c8d09a41d380693f"}
Sep 19 08:28:52.586607 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.586587    2577 generic.go:356] "Generic (PLEG): container finished" podID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerID="4f99ea9a8839cbaa0369dd388a1052956cd8b0e8ae638a93530c7935bfc0a62e" exitCode=0
Sep 19 08:28:52.586704 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.586652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6" event={"ID":"24e9fda7-4baa-4709-80d3-e40c780998f6","Type":"ContainerDied","Data":"4f99ea9a8839cbaa0369dd388a1052956cd8b0e8ae638a93530c7935bfc0a62e"}
Sep 19 08:28:52.586704 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.586680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6" event={"ID":"24e9fda7-4baa-4709-80d3-e40c780998f6","Type":"ContainerStarted","Data":"f09f62d1920446bd576d43a06012f67508d9114f9e4f399def5d49ff4faef81d"}
Sep 19 08:28:52.588169 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.588139    2577 generic.go:356] "Generic (PLEG): container finished" podID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerID="6fd5c25f37ce8788f6c77dbce4f71a59534513456912814b0cbdb77ef59d5f4d" exitCode=0
Sep 19 08:28:52.588259 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.588193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c" event={"ID":"be456d74-a1e7-485d-a144-ddc9436cf2ff","Type":"ContainerDied","Data":"6fd5c25f37ce8788f6c77dbce4f71a59534513456912814b0cbdb77ef59d5f4d"}
Sep 19 08:28:52.588259 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:52.588228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c" event={"ID":"be456d74-a1e7-485d-a144-ddc9436cf2ff","Type":"ContainerStarted","Data":"1a974075663288cbbcbd3f9766157e11e8747915cb0eeed1aed36aad7c3fbccd"}
Sep 19 08:28:53.598239 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:53.598125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6" event={"ID":"24e9fda7-4baa-4709-80d3-e40c780998f6","Type":"ContainerStarted","Data":"1a1d590d2dc541ff4da0afb1f3c6cd820d592be39b2dc099e14cbf09ee296ec3"}
Sep 19 08:28:53.600597 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:53.600517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2" event={"ID":"3a6bc187-4a2e-4520-939f-d39538bbd9d9","Type":"ContainerStarted","Data":"e099ed3cd78e27b92482d0faa0a00137fbfd2e9b9edd74c862abbfec297660b3"}
Sep 19 08:28:54.606636 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:54.606602    2577 generic.go:356] "Generic (PLEG): container finished" podID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerID="fd8d8622c9200fa732ead55b52d555de9d45c260dbc30608cfe5acecfeecdcab" exitCode=0
Sep 19 08:28:54.607111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:54.606682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4" event={"ID":"ba804a76-78b8-4598-a53e-ec7f529bb562","Type":"ContainerDied","Data":"fd8d8622c9200fa732ead55b52d555de9d45c260dbc30608cfe5acecfeecdcab"}
Sep 19 08:28:54.608564 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:54.608545    2577 generic.go:356] "Generic (PLEG): container finished" podID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerID="1a1d590d2dc541ff4da0afb1f3c6cd820d592be39b2dc099e14cbf09ee296ec3" exitCode=0
Sep 19 08:28:54.608667 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:54.608605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6" event={"ID":"24e9fda7-4baa-4709-80d3-e40c780998f6","Type":"ContainerDied","Data":"1a1d590d2dc541ff4da0afb1f3c6cd820d592be39b2dc099e14cbf09ee296ec3"}
Sep 19 08:28:54.610568 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:54.610549    2577 generic.go:356] "Generic (PLEG): container finished" podID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerID="57bf74c58b2a5eddc4d2809806c1bc03cab7d745561602cac37a7f7331e975ca" exitCode=0
Sep 19 08:28:54.610654 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:54.610629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c" event={"ID":"be456d74-a1e7-485d-a144-ddc9436cf2ff","Type":"ContainerDied","Data":"57bf74c58b2a5eddc4d2809806c1bc03cab7d745561602cac37a7f7331e975ca"}
Sep 19 08:28:54.612587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:54.612566    2577 generic.go:356] "Generic (PLEG): container finished" podID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerID="e099ed3cd78e27b92482d0faa0a00137fbfd2e9b9edd74c862abbfec297660b3" exitCode=0
Sep 19 08:28:54.612664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:54.612633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2" event={"ID":"3a6bc187-4a2e-4520-939f-d39538bbd9d9","Type":"ContainerDied","Data":"e099ed3cd78e27b92482d0faa0a00137fbfd2e9b9edd74c862abbfec297660b3"}
Sep 19 08:28:55.619852 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:55.619815    2577 generic.go:356] "Generic (PLEG): container finished" podID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerID="e39e652f7844b7489a10636be6d47e281c0f990e337c69954a930773eb3625f3" exitCode=0
Sep 19 08:28:55.620282 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:55.619882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6" event={"ID":"24e9fda7-4baa-4709-80d3-e40c780998f6","Type":"ContainerDied","Data":"e39e652f7844b7489a10636be6d47e281c0f990e337c69954a930773eb3625f3"}
Sep 19 08:28:55.622164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:55.622144    2577 generic.go:356] "Generic (PLEG): container finished" podID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerID="a619e6e8b7e0c93b9c859d1fedefb1fa6f9b6c8674be14b09d1fe08b1b4a2502" exitCode=0
Sep 19 08:28:55.622277 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:55.622211    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c" event={"ID":"be456d74-a1e7-485d-a144-ddc9436cf2ff","Type":"ContainerDied","Data":"a619e6e8b7e0c93b9c859d1fedefb1fa6f9b6c8674be14b09d1fe08b1b4a2502"}
Sep 19 08:28:55.624389 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:55.624356    2577 generic.go:356] "Generic (PLEG): container finished" podID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerID="3e0a3b366c0cf9c2eda96c536cdc3694b67e3f798a2af715774a4fc70003387a" exitCode=0
Sep 19 08:28:55.624501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:55.624397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2" event={"ID":"3a6bc187-4a2e-4520-939f-d39538bbd9d9","Type":"ContainerDied","Data":"3e0a3b366c0cf9c2eda96c536cdc3694b67e3f798a2af715774a4fc70003387a"}
Sep 19 08:28:55.626558 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:55.626538    2577 generic.go:356] "Generic (PLEG): container finished" podID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerID="e8d7e902544d7ff0fcce3333392c7227fbc909d9ae0c8c9812da3148805705e9" exitCode=0
Sep 19 08:28:55.626640 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:55.626593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4" event={"ID":"ba804a76-78b8-4598-a53e-ec7f529bb562","Type":"ContainerDied","Data":"e8d7e902544d7ff0fcce3333392c7227fbc909d9ae0c8c9812da3148805705e9"}
Sep 19 08:28:56.760372 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.760334    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:56.800661 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.800639    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:56.834679 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.834656    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:56.837964 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.837944    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:56.841113 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841094    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrxsh\" (UniqueName: \"kubernetes.io/projected/be456d74-a1e7-485d-a144-ddc9436cf2ff-kube-api-access-jrxsh\") pod \"be456d74-a1e7-485d-a144-ddc9436cf2ff\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") "
Sep 19 08:28:56.841202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-bundle\") pod \"be456d74-a1e7-485d-a144-ddc9436cf2ff\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") "
Sep 19 08:28:56.841202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841151    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sg2q\" (UniqueName: \"kubernetes.io/projected/3a6bc187-4a2e-4520-939f-d39538bbd9d9-kube-api-access-6sg2q\") pod \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") "
Sep 19 08:28:56.841202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841192    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-util\") pod \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") "
Sep 19 08:28:56.841382 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-util\") pod \"24e9fda7-4baa-4709-80d3-e40c780998f6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") "
Sep 19 08:28:56.841382 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841245    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-bundle\") pod \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\" (UID: \"3a6bc187-4a2e-4520-939f-d39538bbd9d9\") "
Sep 19 08:28:56.841382 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-util\") pod \"be456d74-a1e7-485d-a144-ddc9436cf2ff\" (UID: \"be456d74-a1e7-485d-a144-ddc9436cf2ff\") "
Sep 19 08:28:56.841382 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841316    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gpdm\" (UniqueName: \"kubernetes.io/projected/24e9fda7-4baa-4709-80d3-e40c780998f6-kube-api-access-4gpdm\") pod \"24e9fda7-4baa-4709-80d3-e40c780998f6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") "
Sep 19 08:28:56.841382 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841350    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-bundle\") pod \"24e9fda7-4baa-4709-80d3-e40c780998f6\" (UID: \"24e9fda7-4baa-4709-80d3-e40c780998f6\") "
Sep 19 08:28:56.841967 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841899    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-bundle" (OuterVolumeSpecName: "bundle") pod "be456d74-a1e7-485d-a144-ddc9436cf2ff" (UID: "be456d74-a1e7-485d-a144-ddc9436cf2ff"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:56.841967 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.841900    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-bundle" (OuterVolumeSpecName: "bundle") pod "3a6bc187-4a2e-4520-939f-d39538bbd9d9" (UID: "3a6bc187-4a2e-4520-939f-d39538bbd9d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:56.842499 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.842475    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-bundle" (OuterVolumeSpecName: "bundle") pod "24e9fda7-4baa-4709-80d3-e40c780998f6" (UID: "24e9fda7-4baa-4709-80d3-e40c780998f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:56.845771 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.845745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e9fda7-4baa-4709-80d3-e40c780998f6-kube-api-access-4gpdm" (OuterVolumeSpecName: "kube-api-access-4gpdm") pod "24e9fda7-4baa-4709-80d3-e40c780998f6" (UID: "24e9fda7-4baa-4709-80d3-e40c780998f6"). InnerVolumeSpecName "kube-api-access-4gpdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:28:56.845771 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.845755    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be456d74-a1e7-485d-a144-ddc9436cf2ff-kube-api-access-jrxsh" (OuterVolumeSpecName: "kube-api-access-jrxsh") pod "be456d74-a1e7-485d-a144-ddc9436cf2ff" (UID: "be456d74-a1e7-485d-a144-ddc9436cf2ff"). InnerVolumeSpecName "kube-api-access-jrxsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:28:56.845937 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.845779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6bc187-4a2e-4520-939f-d39538bbd9d9-kube-api-access-6sg2q" (OuterVolumeSpecName: "kube-api-access-6sg2q") pod "3a6bc187-4a2e-4520-939f-d39538bbd9d9" (UID: "3a6bc187-4a2e-4520-939f-d39538bbd9d9"). InnerVolumeSpecName "kube-api-access-6sg2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:28:56.848721 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.848697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-util" (OuterVolumeSpecName: "util") pod "24e9fda7-4baa-4709-80d3-e40c780998f6" (UID: "24e9fda7-4baa-4709-80d3-e40c780998f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:56.850926 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.850903    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-util" (OuterVolumeSpecName: "util") pod "be456d74-a1e7-485d-a144-ddc9436cf2ff" (UID: "be456d74-a1e7-485d-a144-ddc9436cf2ff"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:56.851535 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.851518    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-util" (OuterVolumeSpecName: "util") pod "3a6bc187-4a2e-4520-939f-d39538bbd9d9" (UID: "3a6bc187-4a2e-4520-939f-d39538bbd9d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:56.942441 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942331    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86mrd\" (UniqueName: \"kubernetes.io/projected/ba804a76-78b8-4598-a53e-ec7f529bb562-kube-api-access-86mrd\") pod \"ba804a76-78b8-4598-a53e-ec7f529bb562\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") "
Sep 19 08:28:56.942441 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-util\") pod \"ba804a76-78b8-4598-a53e-ec7f529bb562\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") "
Sep 19 08:28:56.942671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942525    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-bundle\") pod \"ba804a76-78b8-4598-a53e-ec7f529bb562\" (UID: \"ba804a76-78b8-4598-a53e-ec7f529bb562\") "
Sep 19 08:28:56.942761 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942745    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-util\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:56.942817 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942765    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-util\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:56.942817 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942778    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a6bc187-4a2e-4520-939f-d39538bbd9d9-bundle\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:56.942817 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942792    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-util\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:56.942817 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942805    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gpdm\" (UniqueName: \"kubernetes.io/projected/24e9fda7-4baa-4709-80d3-e40c780998f6-kube-api-access-4gpdm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:56.943001 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942819    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24e9fda7-4baa-4709-80d3-e40c780998f6-bundle\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:56.943001 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942833    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrxsh\" (UniqueName: \"kubernetes.io/projected/be456d74-a1e7-485d-a144-ddc9436cf2ff-kube-api-access-jrxsh\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:56.943001 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942846    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be456d74-a1e7-485d-a144-ddc9436cf2ff-bundle\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:56.943001 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942859    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sg2q\" (UniqueName: \"kubernetes.io/projected/3a6bc187-4a2e-4520-939f-d39538bbd9d9-kube-api-access-6sg2q\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:56.943001 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.942917    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-bundle" (OuterVolumeSpecName: "bundle") pod "ba804a76-78b8-4598-a53e-ec7f529bb562" (UID: "ba804a76-78b8-4598-a53e-ec7f529bb562"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:56.944425 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.944403    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba804a76-78b8-4598-a53e-ec7f529bb562-kube-api-access-86mrd" (OuterVolumeSpecName: "kube-api-access-86mrd") pod "ba804a76-78b8-4598-a53e-ec7f529bb562" (UID: "ba804a76-78b8-4598-a53e-ec7f529bb562"). InnerVolumeSpecName "kube-api-access-86mrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:28:56.947710 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:56.947689    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-util" (OuterVolumeSpecName: "util") pod "ba804a76-78b8-4598-a53e-ec7f529bb562" (UID: "ba804a76-78b8-4598-a53e-ec7f529bb562"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:28:57.043585 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.043548    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-bundle\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:57.043585 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.043583    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86mrd\" (UniqueName: \"kubernetes.io/projected/ba804a76-78b8-4598-a53e-ec7f529bb562-kube-api-access-86mrd\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:57.043585 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.043593    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba804a76-78b8-4598-a53e-ec7f529bb562-util\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:28:57.638429 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.638392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2" event={"ID":"3a6bc187-4a2e-4520-939f-d39538bbd9d9","Type":"ContainerDied","Data":"0e843d86e61b6660b621f2442aebcd54425eeed1b7ef6970c1c78a94493c6e8d"}
Sep 19 08:28:57.638429 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.638427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c657tc2"
Sep 19 08:28:57.638658 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.638453    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e843d86e61b6660b621f2442aebcd54425eeed1b7ef6970c1c78a94493c6e8d"
Sep 19 08:28:57.640429 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.640397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4" event={"ID":"ba804a76-78b8-4598-a53e-ec7f529bb562","Type":"ContainerDied","Data":"93302586d2c46a8cc4e43e27dc4d4b276cd49b1c89b7dd34c8d09a41d380693f"}
Sep 19 08:28:57.640585 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.640440    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93302586d2c46a8cc4e43e27dc4d4b276cd49b1c89b7dd34c8d09a41d380693f"
Sep 19 08:28:57.640585 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.640407    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7vhtc4"
Sep 19 08:28:57.642405 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.642386    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6"
Sep 19 08:28:57.642405 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.642390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azwpp6" event={"ID":"24e9fda7-4baa-4709-80d3-e40c780998f6","Type":"ContainerDied","Data":"f09f62d1920446bd576d43a06012f67508d9114f9e4f399def5d49ff4faef81d"}
Sep 19 08:28:57.642588 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.642426    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f09f62d1920446bd576d43a06012f67508d9114f9e4f399def5d49ff4faef81d"
Sep 19 08:28:57.644501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.644478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c" event={"ID":"be456d74-a1e7-485d-a144-ddc9436cf2ff","Type":"ContainerDied","Data":"1a974075663288cbbcbd3f9766157e11e8747915cb0eeed1aed36aad7c3fbccd"}
Sep 19 08:28:57.644585 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.644506    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cz8w9c"
Sep 19 08:28:57.644585 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:28:57.644519    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a974075663288cbbcbd3f9766157e11e8747915cb0eeed1aed36aad7c3fbccd"
Sep 19 08:29:05.050426 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050390    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"]
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050708    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerName="extract"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050719    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerName="extract"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050730    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerName="extract"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050735    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerName="extract"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050742    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerName="extract"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050748    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerName="extract"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050760    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerName="pull"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050765    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerName="pull"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050771    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerName="util"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050775    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerName="util"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050783    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerName="util"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050787    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerName="util"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050792    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerName="extract"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050797    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerName="extract"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050806    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerName="pull"
Sep 19 08:29:05.050804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050811    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerName="pull"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050821    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerName="util"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050826    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerName="util"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050831    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerName="pull"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050836    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerName="pull"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050844    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerName="pull"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050849    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerName="pull"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050855    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerName="util"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050859    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerName="util"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050912    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6bc187-4a2e-4520-939f-d39538bbd9d9" containerName="extract"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050918    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba804a76-78b8-4598-a53e-ec7f529bb562" containerName="extract"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050925    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e9fda7-4baa-4709-80d3-e40c780998f6" containerName="extract"
Sep 19 08:29:05.051249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.050930    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="be456d74-a1e7-485d-a144-ddc9436cf2ff" containerName="extract"
Sep 19 08:29:05.054958 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.054942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"
Sep 19 08:29:05.057756 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.057736    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 19 08:29:05.057962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.057943    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 19 08:29:05.058927 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.058903    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tjkgc\""
Sep 19 08:29:05.064545 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.064521    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"]
Sep 19 08:29:05.105437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.105409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stbjd\" (UniqueName: \"kubernetes.io/projected/b7f4c415-703f-4079-a006-5851c37d5491-kube-api-access-stbjd\") pod \"authorino-operator-6fc8f864f4-9jnzl\" (UID: \"b7f4c415-703f-4079-a006-5851c37d5491\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"
Sep 19 08:29:05.206485 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.206440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stbjd\" (UniqueName: \"kubernetes.io/projected/b7f4c415-703f-4079-a006-5851c37d5491-kube-api-access-stbjd\") pod \"authorino-operator-6fc8f864f4-9jnzl\" (UID: \"b7f4c415-703f-4079-a006-5851c37d5491\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"
Sep 19 08:29:05.234705 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.234675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stbjd\" (UniqueName: \"kubernetes.io/projected/b7f4c415-703f-4079-a006-5851c37d5491-kube-api-access-stbjd\") pod \"authorino-operator-6fc8f864f4-9jnzl\" (UID: \"b7f4c415-703f-4079-a006-5851c37d5491\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"
Sep 19 08:29:05.365401 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.365371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"
Sep 19 08:29:05.641264 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.641239    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"]
Sep 19 08:29:05.643200 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:29:05.643171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f4c415_703f_4079_a006_5851c37d5491.slice/crio-713a401b6fee8f5d05c514fc9b101c58a4bb6bce640ea2582c87751c43eb7d94 WatchSource:0}: Error finding container 713a401b6fee8f5d05c514fc9b101c58a4bb6bce640ea2582c87751c43eb7d94: Status 404 returned error can't find the container with id 713a401b6fee8f5d05c514fc9b101c58a4bb6bce640ea2582c87751c43eb7d94
Sep 19 08:29:05.682181 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:05.682151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl" event={"ID":"b7f4c415-703f-4079-a006-5851c37d5491","Type":"ContainerStarted","Data":"713a401b6fee8f5d05c514fc9b101c58a4bb6bce640ea2582c87751c43eb7d94"}
Sep 19 08:29:07.693528 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:07.693485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl" event={"ID":"b7f4c415-703f-4079-a006-5851c37d5491","Type":"ContainerStarted","Data":"023ba2a3f82bdb8f4556b709661ec3e95822982a6b4f7ba2eee7575bbcebf570"}
Sep 19 08:29:07.693921 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:07.693579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"
Sep 19 08:29:07.715308 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:07.715258    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl" podStartSLOduration=1.428952746 podStartE2EDuration="2.715243585s" totalImagesPullingTime="1.286290839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:29:07.714481238 +0000 UTC m=+515.396064667" watchObservedRunningTime="2026-09-19 08:29:07.715243585 +0000 UTC m=+515.396827017"
Sep 19 08:29:10.574943 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:10.574909    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"]
Sep 19 08:29:10.584149 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:10.584127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"
Sep 19 08:29:10.587548 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:10.587523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pp496\""
Sep 19 08:29:10.589898 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:10.589873    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"]
Sep 19 08:29:10.649831 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:10.649807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vpm\" (UniqueName: \"kubernetes.io/projected/71ac61a2-7f18-42cf-b280-e37a2f2e24e1-kube-api-access-79vpm\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-trd7d\" (UID: \"71ac61a2-7f18-42cf-b280-e37a2f2e24e1\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"
Sep 19 08:29:10.750583 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:10.750554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vpm\" (UniqueName: \"kubernetes.io/projected/71ac61a2-7f18-42cf-b280-e37a2f2e24e1-kube-api-access-79vpm\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-trd7d\" (UID: \"71ac61a2-7f18-42cf-b280-e37a2f2e24e1\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"
Sep 19 08:29:10.767030 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:10.767005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vpm\" (UniqueName: \"kubernetes.io/projected/71ac61a2-7f18-42cf-b280-e37a2f2e24e1-kube-api-access-79vpm\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-trd7d\" (UID: \"71ac61a2-7f18-42cf-b280-e37a2f2e24e1\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"
Sep 19 08:29:10.896301 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:10.896271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"
Sep 19 08:29:11.167926 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:11.167902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"]
Sep 19 08:29:11.170212 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:29:11.170183    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ac61a2_7f18_42cf_b280_e37a2f2e24e1.slice/crio-21c75d4ece9c90ebb80271dd3bbabb78bf53451cbb5f7f379c5ccf1b15380053 WatchSource:0}: Error finding container 21c75d4ece9c90ebb80271dd3bbabb78bf53451cbb5f7f379c5ccf1b15380053: Status 404 returned error can't find the container with id 21c75d4ece9c90ebb80271dd3bbabb78bf53451cbb5f7f379c5ccf1b15380053
Sep 19 08:29:11.714219 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:11.714184    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d" event={"ID":"71ac61a2-7f18-42cf-b280-e37a2f2e24e1","Type":"ContainerStarted","Data":"21c75d4ece9c90ebb80271dd3bbabb78bf53451cbb5f7f379c5ccf1b15380053"}
Sep 19 08:29:13.725938 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:13.725905    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d" event={"ID":"71ac61a2-7f18-42cf-b280-e37a2f2e24e1","Type":"ContainerStarted","Data":"6fc0b71e2459326d6ac6faf3be4eac60b6acb229612940fe3aaba6972b136fb4"}
Sep 19 08:29:13.726427 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:13.726023    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"
Sep 19 08:29:13.744424 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:13.744352    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d" podStartSLOduration=2.160477138 podStartE2EDuration="3.744338639s" totalImagesPullingTime="1.583861501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:29:13.743507391 +0000 UTC m=+521.425090816" watchObservedRunningTime="2026-09-19 08:29:13.744338639 +0000 UTC m=+521.425922072"
Sep 19 08:29:18.701454 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:18.701351    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-9jnzl"
Sep 19 08:29:24.733699 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:29:24.733663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-trd7d"
Sep 19 08:30:06.108132 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.108096    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"]
Sep 19 08:30:06.111662 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.111643    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:06.114166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.114146    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 19 08:30:06.114166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.114160    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d4hsg\""
Sep 19 08:30:06.122816 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.122785    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"]
Sep 19 08:30:06.148173 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.148143    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"]
Sep 19 08:30:06.192043 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.192011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6cr\" (UniqueName: \"kubernetes.io/projected/21f44ad0-93f6-45c6-8d2a-f36c85490def-kube-api-access-7c6cr\") pod \"limitador-limitador-84994f7ffc-6zhpz\" (UID: \"21f44ad0-93f6-45c6-8d2a-f36c85490def\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:06.192166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.192079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/21f44ad0-93f6-45c6-8d2a-f36c85490def-config-file\") pod \"limitador-limitador-84994f7ffc-6zhpz\" (UID: \"21f44ad0-93f6-45c6-8d2a-f36c85490def\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:06.293419 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.293390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/21f44ad0-93f6-45c6-8d2a-f36c85490def-config-file\") pod \"limitador-limitador-84994f7ffc-6zhpz\" (UID: \"21f44ad0-93f6-45c6-8d2a-f36c85490def\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:06.293548 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.293438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6cr\" (UniqueName: \"kubernetes.io/projected/21f44ad0-93f6-45c6-8d2a-f36c85490def-kube-api-access-7c6cr\") pod \"limitador-limitador-84994f7ffc-6zhpz\" (UID: \"21f44ad0-93f6-45c6-8d2a-f36c85490def\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:06.293977 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.293958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/21f44ad0-93f6-45c6-8d2a-f36c85490def-config-file\") pod \"limitador-limitador-84994f7ffc-6zhpz\" (UID: \"21f44ad0-93f6-45c6-8d2a-f36c85490def\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:06.301780 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.301750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6cr\" (UniqueName: \"kubernetes.io/projected/21f44ad0-93f6-45c6-8d2a-f36c85490def-kube-api-access-7c6cr\") pod \"limitador-limitador-84994f7ffc-6zhpz\" (UID: \"21f44ad0-93f6-45c6-8d2a-f36c85490def\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:06.422293 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.422219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:06.697250 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.697215    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"]
Sep 19 08:30:06.698375 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:30:06.698337    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f44ad0_93f6_45c6_8d2a_f36c85490def.slice/crio-379de02762082876df90882f85adb92fd518cd46061b7bb88cade859493bb1f0 WatchSource:0}: Error finding container 379de02762082876df90882f85adb92fd518cd46061b7bb88cade859493bb1f0: Status 404 returned error can't find the container with id 379de02762082876df90882f85adb92fd518cd46061b7bb88cade859493bb1f0
Sep 19 08:30:06.981149 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:06.981070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz" event={"ID":"21f44ad0-93f6-45c6-8d2a-f36c85490def","Type":"ContainerStarted","Data":"379de02762082876df90882f85adb92fd518cd46061b7bb88cade859493bb1f0"}
Sep 19 08:30:07.987942 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:07.987906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz" event={"ID":"21f44ad0-93f6-45c6-8d2a-f36c85490def","Type":"ContainerStarted","Data":"b1876277fbfc303bb81c2a0b9fcef36f00feb1cef57a8361521eec74ba41b411"}
Sep 19 08:30:07.988383 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:07.988054    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:08.007059 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:08.007007    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz" podStartSLOduration=0.918118339 podStartE2EDuration="2.006989894s" totalImagesPullingTime="1.088871555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:30:08.005214687 +0000 UTC m=+575.686798120" watchObservedRunningTime="2026-09-19 08:30:08.006989894 +0000 UTC m=+575.688573326"
Sep 19 08:30:18.993261 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:18.993233    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6zhpz"
Sep 19 08:30:34.444394 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:34.444349    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:30:34.445028 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:34.445004    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:30:51.615031 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.614946    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"]
Sep 19 08:30:51.617570 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.615157    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w" podUID="137e70de-a464-443d-b131-36d4c901c033" containerName="discovery" containerID="cri-o://b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397" gracePeriod=30
Sep 19 08:30:51.861046 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.861019    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:30:51.966936 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.966842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/137e70de-a464-443d-b131-36d4c901c033-local-certs\") pod \"137e70de-a464-443d-b131-36d4c901c033\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") "
Sep 19 08:30:51.966936 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.966891    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-cacerts\") pod \"137e70de-a464-443d-b131-36d4c901c033\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") "
Sep 19 08:30:51.966936 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.966928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/137e70de-a464-443d-b131-36d4c901c033-istio-csr-ca-configmap\") pod \"137e70de-a464-443d-b131-36d4c901c033\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") "
Sep 19 08:30:51.967191 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.966952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxzwz\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-kube-api-access-zxzwz\") pod \"137e70de-a464-443d-b131-36d4c901c033\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") "
Sep 19 08:30:51.967191 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.966985    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-csr-dns-cert\") pod \"137e70de-a464-443d-b131-36d4c901c033\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") "
Sep 19 08:30:51.967191 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.967051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-kubeconfig\") pod \"137e70de-a464-443d-b131-36d4c901c033\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") "
Sep 19 08:30:51.967191 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.967073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-istio-token\") pod \"137e70de-a464-443d-b131-36d4c901c033\" (UID: \"137e70de-a464-443d-b131-36d4c901c033\") "
Sep 19 08:30:51.967438 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.967334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137e70de-a464-443d-b131-36d4c901c033-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "137e70de-a464-443d-b131-36d4c901c033" (UID: "137e70de-a464-443d-b131-36d4c901c033"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:30:51.969329 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.969289    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/137e70de-a464-443d-b131-36d4c901c033-local-certs" (OuterVolumeSpecName: "local-certs") pod "137e70de-a464-443d-b131-36d4c901c033" (UID: "137e70de-a464-443d-b131-36d4c901c033"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:30:51.969508 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.969407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-istio-token" (OuterVolumeSpecName: "istio-token") pod "137e70de-a464-443d-b131-36d4c901c033" (UID: "137e70de-a464-443d-b131-36d4c901c033"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:30:51.969674 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.969650    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-cacerts" (OuterVolumeSpecName: "cacerts") pod "137e70de-a464-443d-b131-36d4c901c033" (UID: "137e70de-a464-443d-b131-36d4c901c033"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:30:51.969770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.969682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-kube-api-access-zxzwz" (OuterVolumeSpecName: "kube-api-access-zxzwz") pod "137e70de-a464-443d-b131-36d4c901c033" (UID: "137e70de-a464-443d-b131-36d4c901c033"). InnerVolumeSpecName "kube-api-access-zxzwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:30:51.969770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.969681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "137e70de-a464-443d-b131-36d4c901c033" (UID: "137e70de-a464-443d-b131-36d4c901c033"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:30:51.969872 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:51.969854    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "137e70de-a464-443d-b131-36d4c901c033" (UID: "137e70de-a464-443d-b131-36d4c901c033"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:30:52.067743 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.067706    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-cacerts\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:30:52.067743 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.067739    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/137e70de-a464-443d-b131-36d4c901c033-istio-csr-ca-configmap\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:30:52.067743 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.067749    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxzwz\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-kube-api-access-zxzwz\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:30:52.067960 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.067760    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-csr-dns-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:30:52.067960 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.067769    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/137e70de-a464-443d-b131-36d4c901c033-istio-kubeconfig\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:30:52.067960 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.067778    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137e70de-a464-443d-b131-36d4c901c033-istio-token\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:30:52.067960 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.067787    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/137e70de-a464-443d-b131-36d4c901c033-local-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:30:52.208288 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.208253    2577 generic.go:356] "Generic (PLEG): container finished" podID="137e70de-a464-443d-b131-36d4c901c033" containerID="b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397" exitCode=0
Sep 19 08:30:52.208481 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.208303    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"
Sep 19 08:30:52.208481 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.208302    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w" event={"ID":"137e70de-a464-443d-b131-36d4c901c033","Type":"ContainerDied","Data":"b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397"}
Sep 19 08:30:52.208481 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.208417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w" event={"ID":"137e70de-a464-443d-b131-36d4c901c033","Type":"ContainerDied","Data":"e48e63858b266f72552f200fea9cd102ae3d77b85c22a112ea1ca1a2d31defec"}
Sep 19 08:30:52.208481 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.208444    2577 scope.go:117] "RemoveContainer" containerID="b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397"
Sep 19 08:30:52.217770 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.217743    2577 scope.go:117] "RemoveContainer" containerID="b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397"
Sep 19 08:30:52.218051 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:30:52.218031    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397\": container with ID starting with b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397 not found: ID does not exist" containerID="b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397"
Sep 19 08:30:52.218104 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.218059    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397"} err="failed to get container status \"b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397\": rpc error: code = NotFound desc = could not find container \"b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397\": container with ID starting with b0ae0d74273e1f9b5533942e2962cd2f38bc22ac67534acd3448f0a1ae3b3397 not found: ID does not exist"
Sep 19 08:30:52.233151 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.233121    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"]
Sep 19 08:30:52.237019 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.236996    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kvm9w"]
Sep 19 08:30:52.503717 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:52.503635    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137e70de-a464-443d-b131-36d4c901c033" path="/var/lib/kubelet/pods/137e70de-a464-443d-b131-36d4c901c033/volumes"
Sep 19 08:30:58.133798 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.133764    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-ds5w5"]
Sep 19 08:30:58.134198 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.134105    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137e70de-a464-443d-b131-36d4c901c033" containerName="discovery"
Sep 19 08:30:58.134198 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.134116    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="137e70de-a464-443d-b131-36d4c901c033" containerName="discovery"
Sep 19 08:30:58.134198 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.134168    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="137e70de-a464-443d-b131-36d4c901c033" containerName="discovery"
Sep 19 08:30:58.138515 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.138490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:30:58.141620 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.141599    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 19 08:30:58.142512 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.142488    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 19 08:30:58.142623 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.142512    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 19 08:30:58.142623 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.142519    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-gdmqf\""
Sep 19 08:30:58.147314 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.147279    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-ds5w5"]
Sep 19 08:30:58.149457 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.149434    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6888f9f64c-f28sc"]
Sep 19 08:30:58.152870 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.152849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:30:58.155505 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.155483    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-qdkw6\""
Sep 19 08:30:58.155624 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.155488    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 19 08:30:58.163274 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.163251    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6888f9f64c-f28sc"]
Sep 19 08:30:58.226378 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.226333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpfj2\" (UniqueName: \"kubernetes.io/projected/54bc0708-6c78-4d4a-9c02-e24c5652331e-kube-api-access-dpfj2\") pod \"kserve-controller-manager-7dcf4bff6-ds5w5\" (UID: \"54bc0708-6c78-4d4a-9c02-e24c5652331e\") " pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:30:58.226558 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.226462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zzs2\" (UniqueName: \"kubernetes.io/projected/3479dbd7-c751-4d4a-9e75-985e0a7c3c15-kube-api-access-5zzs2\") pod \"llmisvc-controller-manager-6888f9f64c-f28sc\" (UID: \"3479dbd7-c751-4d4a-9e75-985e0a7c3c15\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:30:58.226558 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.226504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3479dbd7-c751-4d4a-9e75-985e0a7c3c15-cert\") pod \"llmisvc-controller-manager-6888f9f64c-f28sc\" (UID: \"3479dbd7-c751-4d4a-9e75-985e0a7c3c15\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:30:58.226558 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.226538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54bc0708-6c78-4d4a-9c02-e24c5652331e-cert\") pod \"kserve-controller-manager-7dcf4bff6-ds5w5\" (UID: \"54bc0708-6c78-4d4a-9c02-e24c5652331e\") " pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:30:58.327120 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.327083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zzs2\" (UniqueName: \"kubernetes.io/projected/3479dbd7-c751-4d4a-9e75-985e0a7c3c15-kube-api-access-5zzs2\") pod \"llmisvc-controller-manager-6888f9f64c-f28sc\" (UID: \"3479dbd7-c751-4d4a-9e75-985e0a7c3c15\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:30:58.327120 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.327124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3479dbd7-c751-4d4a-9e75-985e0a7c3c15-cert\") pod \"llmisvc-controller-manager-6888f9f64c-f28sc\" (UID: \"3479dbd7-c751-4d4a-9e75-985e0a7c3c15\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:30:58.327415 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.327169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54bc0708-6c78-4d4a-9c02-e24c5652331e-cert\") pod \"kserve-controller-manager-7dcf4bff6-ds5w5\" (UID: \"54bc0708-6c78-4d4a-9c02-e24c5652331e\") " pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:30:58.327415 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.327197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpfj2\" (UniqueName: \"kubernetes.io/projected/54bc0708-6c78-4d4a-9c02-e24c5652331e-kube-api-access-dpfj2\") pod \"kserve-controller-manager-7dcf4bff6-ds5w5\" (UID: \"54bc0708-6c78-4d4a-9c02-e24c5652331e\") " pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:30:58.329728 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.329697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3479dbd7-c751-4d4a-9e75-985e0a7c3c15-cert\") pod \"llmisvc-controller-manager-6888f9f64c-f28sc\" (UID: \"3479dbd7-c751-4d4a-9e75-985e0a7c3c15\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:30:58.329831 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.329703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54bc0708-6c78-4d4a-9c02-e24c5652331e-cert\") pod \"kserve-controller-manager-7dcf4bff6-ds5w5\" (UID: \"54bc0708-6c78-4d4a-9c02-e24c5652331e\") " pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:30:58.335508 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.335476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zzs2\" (UniqueName: \"kubernetes.io/projected/3479dbd7-c751-4d4a-9e75-985e0a7c3c15-kube-api-access-5zzs2\") pod \"llmisvc-controller-manager-6888f9f64c-f28sc\" (UID: \"3479dbd7-c751-4d4a-9e75-985e0a7c3c15\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:30:58.335508 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.335502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpfj2\" (UniqueName: \"kubernetes.io/projected/54bc0708-6c78-4d4a-9c02-e24c5652331e-kube-api-access-dpfj2\") pod \"kserve-controller-manager-7dcf4bff6-ds5w5\" (UID: \"54bc0708-6c78-4d4a-9c02-e24c5652331e\") " pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:30:58.450596 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.450506    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:30:58.464718 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.464693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:30:58.722218 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.722194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-ds5w5"]
Sep 19 08:30:58.723993 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:30:58.723963    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bc0708_6c78_4d4a_9c02_e24c5652331e.slice/crio-847e060e5139b48a737970c1680100f4191397eb220210c613f8f9f3aeae22fd WatchSource:0}: Error finding container 847e060e5139b48a737970c1680100f4191397eb220210c613f8f9f3aeae22fd: Status 404 returned error can't find the container with id 847e060e5139b48a737970c1680100f4191397eb220210c613f8f9f3aeae22fd
Sep 19 08:30:58.725554 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.725533    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:30:58.747388 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:58.747348    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6888f9f64c-f28sc"]
Sep 19 08:30:58.749068 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:30:58.749044    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3479dbd7_c751_4d4a_9e75_985e0a7c3c15.slice/crio-4e5045e69358b6d6d31ad22aeb8600e7919aa0d6aa885145f952816e9e44e005 WatchSource:0}: Error finding container 4e5045e69358b6d6d31ad22aeb8600e7919aa0d6aa885145f952816e9e44e005: Status 404 returned error can't find the container with id 4e5045e69358b6d6d31ad22aeb8600e7919aa0d6aa885145f952816e9e44e005
Sep 19 08:30:59.246108 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:59.246072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5" event={"ID":"54bc0708-6c78-4d4a-9c02-e24c5652331e","Type":"ContainerStarted","Data":"847e060e5139b48a737970c1680100f4191397eb220210c613f8f9f3aeae22fd"}
Sep 19 08:30:59.247275 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:30:59.247251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc" event={"ID":"3479dbd7-c751-4d4a-9e75-985e0a7c3c15","Type":"ContainerStarted","Data":"4e5045e69358b6d6d31ad22aeb8600e7919aa0d6aa885145f952816e9e44e005"}
Sep 19 08:31:03.278094 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:03.278054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc" event={"ID":"3479dbd7-c751-4d4a-9e75-985e0a7c3c15","Type":"ContainerStarted","Data":"6f717818380af560ea12bd49dc30b0f917cdf13193191bc5ec5625e80852c97a"}
Sep 19 08:31:03.278585 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:03.278114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:31:03.279507 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:03.279482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5" event={"ID":"54bc0708-6c78-4d4a-9c02-e24c5652331e","Type":"ContainerStarted","Data":"87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af"}
Sep 19 08:31:03.279627 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:03.279610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:31:03.295985 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:03.295937    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc" podStartSLOduration=1.874191524 podStartE2EDuration="5.295922174s" totalImagesPullingTime="3.42173065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:03.294756154 +0000 UTC m=+630.976339587" watchObservedRunningTime="2026-09-19 08:31:03.295922174 +0000 UTC m=+630.977505601"
Sep 19 08:31:03.316483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:03.316435    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5" podStartSLOduration=1.8730278870000001 podStartE2EDuration="5.316418509s" totalImagesPullingTime="3.443390622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:03.316233427 +0000 UTC m=+630.997816860" watchObservedRunningTime="2026-09-19 08:31:03.316418509 +0000 UTC m=+630.998001942"
Sep 19 08:31:34.287519 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:34.287488    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6888f9f64c-f28sc"
Sep 19 08:31:34.290271 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:34.290254    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:31:41.444554 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.444515    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-ds5w5"]
Sep 19 08:31:41.445021 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.444701    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5" podUID="54bc0708-6c78-4d4a-9c02-e24c5652331e" containerName="manager" containerID="cri-o://87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af" gracePeriod=10
Sep 19 08:31:41.466895 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.466870    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-7f7nl"]
Sep 19 08:31:41.470204 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.470184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:31:41.479299 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.479260    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-7f7nl"]
Sep 19 08:31:41.574323 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.574292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6v7\" (UniqueName: \"kubernetes.io/projected/04c6aeb1-8f45-4595-a064-c3888a62c0a3-kube-api-access-2n6v7\") pod \"kserve-controller-manager-7dcf4bff6-7f7nl\" (UID: \"04c6aeb1-8f45-4595-a064-c3888a62c0a3\") " pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:31:41.574458 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.574346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04c6aeb1-8f45-4595-a064-c3888a62c0a3-cert\") pod \"kserve-controller-manager-7dcf4bff6-7f7nl\" (UID: \"04c6aeb1-8f45-4595-a064-c3888a62c0a3\") " pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:31:41.675163 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.675138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6v7\" (UniqueName: \"kubernetes.io/projected/04c6aeb1-8f45-4595-a064-c3888a62c0a3-kube-api-access-2n6v7\") pod \"kserve-controller-manager-7dcf4bff6-7f7nl\" (UID: \"04c6aeb1-8f45-4595-a064-c3888a62c0a3\") " pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:31:41.675273 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.675181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04c6aeb1-8f45-4595-a064-c3888a62c0a3-cert\") pod \"kserve-controller-manager-7dcf4bff6-7f7nl\" (UID: \"04c6aeb1-8f45-4595-a064-c3888a62c0a3\") " pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:31:41.677358 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.677332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04c6aeb1-8f45-4595-a064-c3888a62c0a3-cert\") pod \"kserve-controller-manager-7dcf4bff6-7f7nl\" (UID: \"04c6aeb1-8f45-4595-a064-c3888a62c0a3\") " pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:31:41.684134 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.684110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6v7\" (UniqueName: \"kubernetes.io/projected/04c6aeb1-8f45-4595-a064-c3888a62c0a3-kube-api-access-2n6v7\") pod \"kserve-controller-manager-7dcf4bff6-7f7nl\" (UID: \"04c6aeb1-8f45-4595-a064-c3888a62c0a3\") " pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:31:41.700795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.700747    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:31:41.776656 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.776618    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpfj2\" (UniqueName: \"kubernetes.io/projected/54bc0708-6c78-4d4a-9c02-e24c5652331e-kube-api-access-dpfj2\") pod \"54bc0708-6c78-4d4a-9c02-e24c5652331e\" (UID: \"54bc0708-6c78-4d4a-9c02-e24c5652331e\") "
Sep 19 08:31:41.776837 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.776702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54bc0708-6c78-4d4a-9c02-e24c5652331e-cert\") pod \"54bc0708-6c78-4d4a-9c02-e24c5652331e\" (UID: \"54bc0708-6c78-4d4a-9c02-e24c5652331e\") "
Sep 19 08:31:41.778789 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.778762    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bc0708-6c78-4d4a-9c02-e24c5652331e-kube-api-access-dpfj2" (OuterVolumeSpecName: "kube-api-access-dpfj2") pod "54bc0708-6c78-4d4a-9c02-e24c5652331e" (UID: "54bc0708-6c78-4d4a-9c02-e24c5652331e"). InnerVolumeSpecName "kube-api-access-dpfj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:31:41.778897 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.778786    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54bc0708-6c78-4d4a-9c02-e24c5652331e-cert" (OuterVolumeSpecName: "cert") pod "54bc0708-6c78-4d4a-9c02-e24c5652331e" (UID: "54bc0708-6c78-4d4a-9c02-e24c5652331e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:31:41.822747 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.822726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:31:41.878027 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.877996    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpfj2\" (UniqueName: \"kubernetes.io/projected/54bc0708-6c78-4d4a-9c02-e24c5652331e-kube-api-access-dpfj2\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:31:41.878121 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:41.878033    2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54bc0708-6c78-4d4a-9c02-e24c5652331e-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:31:42.091245 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.091219    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-7f7nl"]
Sep 19 08:31:42.094590 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:31:42.094561    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c6aeb1_8f45_4595_a064_c3888a62c0a3.slice/crio-cde5faf2d1d1a34d7c54a29dfa735e23fd47569bbf8072a8ad8820cd1b22bd85 WatchSource:0}: Error finding container cde5faf2d1d1a34d7c54a29dfa735e23fd47569bbf8072a8ad8820cd1b22bd85: Status 404 returned error can't find the container with id cde5faf2d1d1a34d7c54a29dfa735e23fd47569bbf8072a8ad8820cd1b22bd85
Sep 19 08:31:42.488726 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.488694    2577 generic.go:356] "Generic (PLEG): container finished" podID="54bc0708-6c78-4d4a-9c02-e24c5652331e" containerID="87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af" exitCode=0
Sep 19 08:31:42.488726 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.488729    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5"
Sep 19 08:31:42.489176 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.488785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5" event={"ID":"54bc0708-6c78-4d4a-9c02-e24c5652331e","Type":"ContainerDied","Data":"87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af"}
Sep 19 08:31:42.489176 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.488846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-ds5w5" event={"ID":"54bc0708-6c78-4d4a-9c02-e24c5652331e","Type":"ContainerDied","Data":"847e060e5139b48a737970c1680100f4191397eb220210c613f8f9f3aeae22fd"}
Sep 19 08:31:42.489176 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.488878    2577 scope.go:117] "RemoveContainer" containerID="87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af"
Sep 19 08:31:42.490673 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.490651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl" event={"ID":"04c6aeb1-8f45-4595-a064-c3888a62c0a3","Type":"ContainerStarted","Data":"0e1fb1c73c687699bc574cdfbdd1262f62d0f51467d700559541a20acc167655"}
Sep 19 08:31:42.490767 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.490689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl" event={"ID":"04c6aeb1-8f45-4595-a064-c3888a62c0a3","Type":"ContainerStarted","Data":"cde5faf2d1d1a34d7c54a29dfa735e23fd47569bbf8072a8ad8820cd1b22bd85"}
Sep 19 08:31:42.490767 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.490716    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:31:42.497927 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.497911    2577 scope.go:117] "RemoveContainer" containerID="87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af"
Sep 19 08:31:42.498192 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:31:42.498163    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af\": container with ID starting with 87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af not found: ID does not exist" containerID="87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af"
Sep 19 08:31:42.498264 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.498203    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af"} err="failed to get container status \"87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af\": rpc error: code = NotFound desc = could not find container \"87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af\": container with ID starting with 87a61bf9c673b2665dca4cdd80d20209e4efd22e8e3dd087c255d32a30f5a3af not found: ID does not exist"
Sep 19 08:31:42.513117 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.513070    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl" podStartSLOduration=1.200066786 podStartE2EDuration="1.513055389s" totalImagesPullingTime="312.988603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:42.510269622 +0000 UTC m=+670.191853055" watchObservedRunningTime="2026-09-19 08:31:42.513055389 +0000 UTC m=+670.194638822"
Sep 19 08:31:42.524664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.524639    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-ds5w5"]
Sep 19 08:31:42.528741 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:42.528718    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7dcf4bff6-ds5w5"]
Sep 19 08:31:44.504612 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:31:44.504580    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54bc0708-6c78-4d4a-9c02-e24c5652331e" path="/var/lib/kubelet/pods/54bc0708-6c78-4d4a-9c02-e24c5652331e/volumes"
Sep 19 08:32:13.501812 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:13.501779    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7dcf4bff6-7f7nl"
Sep 19 08:32:14.834985 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.834954    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-2wjrp"]
Sep 19 08:32:14.835500 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.835481    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54bc0708-6c78-4d4a-9c02-e24c5652331e" containerName="manager"
Sep 19 08:32:14.835587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.835503    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="54bc0708-6c78-4d4a-9c02-e24c5652331e" containerName="manager"
Sep 19 08:32:14.835643 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.835633    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="54bc0708-6c78-4d4a-9c02-e24c5652331e" containerName="manager"
Sep 19 08:32:14.838798 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.838772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:14.841376 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.841346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-z4qlm\""
Sep 19 08:32:14.841619 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.841602    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 19 08:32:14.847855 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.847833    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-f76dd"]
Sep 19 08:32:14.852569 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.852552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:14.853326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.853300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2wjrp"]
Sep 19 08:32:14.871844 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.871819    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 19 08:32:14.871979 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.871906    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 19 08:32:14.872134 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.872114    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-gglt9\""
Sep 19 08:32:14.872211 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.872126    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-f76dd"]
Sep 19 08:32:14.973488 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.973454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctwsq\" (UniqueName: \"kubernetes.io/projected/3a837fe5-4583-41b4-828f-6b3bc09a1354-kube-api-access-ctwsq\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:14.973647 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.973502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c99ff65-cffa-4499-b71a-a418c9b2f7bd-tls-certs\") pod \"model-serving-api-7b954ffc95-2wjrp\" (UID: \"3c99ff65-cffa-4499-b71a-a418c9b2f7bd\") " pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:14.973647 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.973529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a837fe5-4583-41b4-828f-6b3bc09a1354-metrics-certs\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:14.973647 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.973619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6z2\" (UniqueName: \"kubernetes.io/projected/3c99ff65-cffa-4499-b71a-a418c9b2f7bd-kube-api-access-td6z2\") pod \"model-serving-api-7b954ffc95-2wjrp\" (UID: \"3c99ff65-cffa-4499-b71a-a418c9b2f7bd\") " pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:14.973753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:14.973650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a837fe5-4583-41b4-828f-6b3bc09a1354-cert\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:15.074442 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.074400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c99ff65-cffa-4499-b71a-a418c9b2f7bd-tls-certs\") pod \"model-serving-api-7b954ffc95-2wjrp\" (UID: \"3c99ff65-cffa-4499-b71a-a418c9b2f7bd\") " pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:15.074649 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.074460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a837fe5-4583-41b4-828f-6b3bc09a1354-metrics-certs\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:15.074649 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:32:15.074478    2577 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 19 08:32:15.074649 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:32:15.074553    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c99ff65-cffa-4499-b71a-a418c9b2f7bd-tls-certs podName:3c99ff65-cffa-4499-b71a-a418c9b2f7bd nodeName:}" failed. No retries permitted until 2026-09-19 08:32:15.574530911 +0000 UTC m=+703.256114321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3c99ff65-cffa-4499-b71a-a418c9b2f7bd-tls-certs") pod "model-serving-api-7b954ffc95-2wjrp" (UID: "3c99ff65-cffa-4499-b71a-a418c9b2f7bd") : secret "model-serving-api-tls" not found
Sep 19 08:32:15.074649 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.074575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6z2\" (UniqueName: \"kubernetes.io/projected/3c99ff65-cffa-4499-b71a-a418c9b2f7bd-kube-api-access-td6z2\") pod \"model-serving-api-7b954ffc95-2wjrp\" (UID: \"3c99ff65-cffa-4499-b71a-a418c9b2f7bd\") " pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:15.074649 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.074612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a837fe5-4583-41b4-828f-6b3bc09a1354-cert\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:15.074913 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.074664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctwsq\" (UniqueName: \"kubernetes.io/projected/3a837fe5-4583-41b4-828f-6b3bc09a1354-kube-api-access-ctwsq\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:15.074913 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:32:15.074812    2577 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 19 08:32:15.074913 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:32:15.074881    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a837fe5-4583-41b4-828f-6b3bc09a1354-cert podName:3a837fe5-4583-41b4-828f-6b3bc09a1354 nodeName:}" failed. No retries permitted until 2026-09-19 08:32:15.574862041 +0000 UTC m=+703.256445453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3a837fe5-4583-41b4-828f-6b3bc09a1354-cert") pod "odh-model-controller-7b966d7984-f76dd" (UID: "3a837fe5-4583-41b4-828f-6b3bc09a1354") : secret "odh-model-controller-webhook-cert" not found
Sep 19 08:32:15.076892 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.076875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a837fe5-4583-41b4-828f-6b3bc09a1354-metrics-certs\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:15.084959 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.084938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6z2\" (UniqueName: \"kubernetes.io/projected/3c99ff65-cffa-4499-b71a-a418c9b2f7bd-kube-api-access-td6z2\") pod \"model-serving-api-7b954ffc95-2wjrp\" (UID: \"3c99ff65-cffa-4499-b71a-a418c9b2f7bd\") " pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:15.087824 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.087763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctwsq\" (UniqueName: \"kubernetes.io/projected/3a837fe5-4583-41b4-828f-6b3bc09a1354-kube-api-access-ctwsq\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:15.577326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.577280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a837fe5-4583-41b4-828f-6b3bc09a1354-cert\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:15.577526 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.577399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c99ff65-cffa-4499-b71a-a418c9b2f7bd-tls-certs\") pod \"model-serving-api-7b954ffc95-2wjrp\" (UID: \"3c99ff65-cffa-4499-b71a-a418c9b2f7bd\") " pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:15.579603 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.579578    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a837fe5-4583-41b4-828f-6b3bc09a1354-cert\") pod \"odh-model-controller-7b966d7984-f76dd\" (UID: \"3a837fe5-4583-41b4-828f-6b3bc09a1354\") " pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:15.579758 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.579734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c99ff65-cffa-4499-b71a-a418c9b2f7bd-tls-certs\") pod \"model-serving-api-7b954ffc95-2wjrp\" (UID: \"3c99ff65-cffa-4499-b71a-a418c9b2f7bd\") " pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:15.764547 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.764520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:15.784473 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:15.784442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:16.038235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:16.038209    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2wjrp"]
Sep 19 08:32:16.040168 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:32:16.040130    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c99ff65_cffa_4499_b71a_a418c9b2f7bd.slice/crio-87829f477c1be43eed8773a67f262bbb1bd61c882c64e9b0526f252da8a34ffb WatchSource:0}: Error finding container 87829f477c1be43eed8773a67f262bbb1bd61c882c64e9b0526f252da8a34ffb: Status 404 returned error can't find the container with id 87829f477c1be43eed8773a67f262bbb1bd61c882c64e9b0526f252da8a34ffb
Sep 19 08:32:16.060792 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:16.060772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-f76dd"]
Sep 19 08:32:16.061884 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:32:16.061862    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a837fe5_4583_41b4_828f_6b3bc09a1354.slice/crio-e2d7cdb18c839ed1241a3d024b9e1b185af0f95a1706f0805a6cd45c9318d9a1 WatchSource:0}: Error finding container e2d7cdb18c839ed1241a3d024b9e1b185af0f95a1706f0805a6cd45c9318d9a1: Status 404 returned error can't find the container with id e2d7cdb18c839ed1241a3d024b9e1b185af0f95a1706f0805a6cd45c9318d9a1
Sep 19 08:32:16.672964 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:16.672914    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2wjrp" event={"ID":"3c99ff65-cffa-4499-b71a-a418c9b2f7bd","Type":"ContainerStarted","Data":"87829f477c1be43eed8773a67f262bbb1bd61c882c64e9b0526f252da8a34ffb"}
Sep 19 08:32:16.674839 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:16.674807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-f76dd" event={"ID":"3a837fe5-4583-41b4-828f-6b3bc09a1354","Type":"ContainerStarted","Data":"e2d7cdb18c839ed1241a3d024b9e1b185af0f95a1706f0805a6cd45c9318d9a1"}
Sep 19 08:32:19.694174 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:19.694134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2wjrp" event={"ID":"3c99ff65-cffa-4499-b71a-a418c9b2f7bd","Type":"ContainerStarted","Data":"7c750b63d6e83b7f97d5a7954c4a41f974c1b0871d98f100e6c6c6b83806f3d3"}
Sep 19 08:32:19.694645 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:19.694282    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:19.695702 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:19.695678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-f76dd" event={"ID":"3a837fe5-4583-41b4-828f-6b3bc09a1354","Type":"ContainerStarted","Data":"e4f59e1514cbae7eed021fd42b753f904ad23d3462725fcff4ecb3a16515b703"}
Sep 19 08:32:19.695810 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:19.695722    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:19.710061 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:19.710014    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-2wjrp" podStartSLOduration=3.015719678 podStartE2EDuration="5.710001369s" totalImagesPullingTime="2.694281691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:32:19.709120084 +0000 UTC m=+707.390703521" watchObservedRunningTime="2026-09-19 08:32:19.710001369 +0000 UTC m=+707.391584804"
Sep 19 08:32:19.729491 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:19.729446    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-f76dd" podStartSLOduration=3.004625453 podStartE2EDuration="5.729432467s" totalImagesPullingTime="2.724807014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:32:19.727092937 +0000 UTC m=+707.408676380" watchObservedRunningTime="2026-09-19 08:32:19.729432467 +0000 UTC m=+707.411015900"
Sep 19 08:32:30.704017 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:30.703984    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-f76dd"
Sep 19 08:32:30.705897 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:30.705878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-2wjrp"
Sep 19 08:32:31.535639 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.535601    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-9kttj"]
Sep 19 08:32:31.540079 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.540062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9kttj"
Sep 19 08:32:31.542494 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.542471    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 19 08:32:31.542637 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.542498    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fqpnd\""
Sep 19 08:32:31.542637 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.542498    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 19 08:32:31.546472 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.546447    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9kttj"]
Sep 19 08:32:31.596784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.596753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbptx\" (UniqueName: \"kubernetes.io/projected/8a33e212-a14a-455d-b751-e7ae18d8a7fa-kube-api-access-xbptx\") pod \"s3-init-9kttj\" (UID: \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\") " pod="kserve/s3-init-9kttj"
Sep 19 08:32:31.596938 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.596797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a33e212-a14a-455d-b751-e7ae18d8a7fa-model-cache\") pod \"s3-init-9kttj\" (UID: \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\") " pod="kserve/s3-init-9kttj"
Sep 19 08:32:31.698755 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.698264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbptx\" (UniqueName: \"kubernetes.io/projected/8a33e212-a14a-455d-b751-e7ae18d8a7fa-kube-api-access-xbptx\") pod \"s3-init-9kttj\" (UID: \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\") " pod="kserve/s3-init-9kttj"
Sep 19 08:32:31.698755 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.698335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a33e212-a14a-455d-b751-e7ae18d8a7fa-model-cache\") pod \"s3-init-9kttj\" (UID: \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\") " pod="kserve/s3-init-9kttj"
Sep 19 08:32:31.703295 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.699017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a33e212-a14a-455d-b751-e7ae18d8a7fa-model-cache\") pod \"s3-init-9kttj\" (UID: \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\") " pod="kserve/s3-init-9kttj"
Sep 19 08:32:31.706913 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.706886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbptx\" (UniqueName: \"kubernetes.io/projected/8a33e212-a14a-455d-b751-e7ae18d8a7fa-kube-api-access-xbptx\") pod \"s3-init-9kttj\" (UID: \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\") " pod="kserve/s3-init-9kttj"
Sep 19 08:32:31.849927 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:31.849901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9kttj"
Sep 19 08:32:32.120749 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:32.120724    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-9kttj"]
Sep 19 08:32:32.123095 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:32:32.123062    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a33e212_a14a_455d_b751_e7ae18d8a7fa.slice/crio-c7bd50b590a3683588471fd55065513f304b56a67a320d9002f61aef229b48e6 WatchSource:0}: Error finding container c7bd50b590a3683588471fd55065513f304b56a67a320d9002f61aef229b48e6: Status 404 returned error can't find the container with id c7bd50b590a3683588471fd55065513f304b56a67a320d9002f61aef229b48e6
Sep 19 08:32:32.766041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:32.765999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9kttj" event={"ID":"8a33e212-a14a-455d-b751-e7ae18d8a7fa","Type":"ContainerStarted","Data":"c7bd50b590a3683588471fd55065513f304b56a67a320d9002f61aef229b48e6"}
Sep 19 08:32:35.560586 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:35.560562    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 19 08:32:35.784669 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:35.784574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9kttj" event={"ID":"8a33e212-a14a-455d-b751-e7ae18d8a7fa","Type":"ContainerStarted","Data":"a1c253b9f4f3ab0a8601763a29c84d7fe504d945471bf6a64c8d177f7fb7a88f"}
Sep 19 08:32:37.796866 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:37.796781    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a33e212-a14a-455d-b751-e7ae18d8a7fa" containerID="a1c253b9f4f3ab0a8601763a29c84d7fe504d945471bf6a64c8d177f7fb7a88f" exitCode=0
Sep 19 08:32:37.796866 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:37.796852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9kttj" event={"ID":"8a33e212-a14a-455d-b751-e7ae18d8a7fa","Type":"ContainerDied","Data":"a1c253b9f4f3ab0a8601763a29c84d7fe504d945471bf6a64c8d177f7fb7a88f"}
Sep 19 08:32:42.229044 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:42.229020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 19 08:32:42.828867 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:42.828830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9kttj" event={"ID":"8a33e212-a14a-455d-b751-e7ae18d8a7fa","Type":"ContainerStarted","Data":"f495c18e1134e27de5d5c020a93b97fecde14c99d3f55502d396a2188c58c9b8"}
Sep 19 08:32:42.847886 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:32:42.847834    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-9kttj" podStartSLOduration=2.144982021 podStartE2EDuration="11.847815724s" totalImagesPullingTime="7.861405178s" totalInitContainerRuntime="1.841428525s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:32:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:32:42.845766491 +0000 UTC m=+730.527349927" watchObservedRunningTime="2026-09-19 08:32:42.847815724 +0000 UTC m=+730.529399157"
Sep 19 08:33:48.164452 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:48.164420    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a33e212-a14a-455d-b751-e7ae18d8a7fa" containerID="f495c18e1134e27de5d5c020a93b97fecde14c99d3f55502d396a2188c58c9b8" exitCode=0
Sep 19 08:33:48.164863 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:48.164478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9kttj" event={"ID":"8a33e212-a14a-455d-b751-e7ae18d8a7fa","Type":"ContainerDied","Data":"f495c18e1134e27de5d5c020a93b97fecde14c99d3f55502d396a2188c58c9b8"}
Sep 19 08:33:49.301465 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:49.301439    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9kttj"
Sep 19 08:33:49.445664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:49.445570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbptx\" (UniqueName: \"kubernetes.io/projected/8a33e212-a14a-455d-b751-e7ae18d8a7fa-kube-api-access-xbptx\") pod \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\" (UID: \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\") "
Sep 19 08:33:49.445664 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:49.445612    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a33e212-a14a-455d-b751-e7ae18d8a7fa-model-cache\") pod \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\" (UID: \"8a33e212-a14a-455d-b751-e7ae18d8a7fa\") "
Sep 19 08:33:49.447706 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:49.447673    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a33e212-a14a-455d-b751-e7ae18d8a7fa-kube-api-access-xbptx" (OuterVolumeSpecName: "kube-api-access-xbptx") pod "8a33e212-a14a-455d-b751-e7ae18d8a7fa" (UID: "8a33e212-a14a-455d-b751-e7ae18d8a7fa"). InnerVolumeSpecName "kube-api-access-xbptx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:33:49.464318 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:49.464291    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a33e212-a14a-455d-b751-e7ae18d8a7fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a33e212-a14a-455d-b751-e7ae18d8a7fa" (UID: "8a33e212-a14a-455d-b751-e7ae18d8a7fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:33:49.546765 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:49.546732    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbptx\" (UniqueName: \"kubernetes.io/projected/8a33e212-a14a-455d-b751-e7ae18d8a7fa-kube-api-access-xbptx\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:33:49.546765 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:49.546760    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a33e212-a14a-455d-b751-e7ae18d8a7fa-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:33:50.175830 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:50.175801    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-9kttj"
Sep 19 08:33:50.176033 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:50.175839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-9kttj" event={"ID":"8a33e212-a14a-455d-b751-e7ae18d8a7fa","Type":"ContainerDied","Data":"c7bd50b590a3683588471fd55065513f304b56a67a320d9002f61aef229b48e6"}
Sep 19 08:33:50.176033 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:33:50.175872    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7bd50b590a3683588471fd55065513f304b56a67a320d9002f61aef229b48e6"
Sep 19 08:34:13.392526 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.392489    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"]
Sep 19 08:34:13.393090 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.393070    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a33e212-a14a-455d-b751-e7ae18d8a7fa" containerName="s3-init"
Sep 19 08:34:13.393177 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.393095    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a33e212-a14a-455d-b751-e7ae18d8a7fa" containerName="s3-init"
Sep 19 08:34:13.393177 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.393124    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a33e212-a14a-455d-b751-e7ae18d8a7fa" containerName="download-hf-model"
Sep 19 08:34:13.393177 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.393133    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a33e212-a14a-455d-b751-e7ae18d8a7fa" containerName="download-hf-model"
Sep 19 08:34:13.393843 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.393816    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a33e212-a14a-455d-b751-e7ae18d8a7fa" containerName="s3-init"
Sep 19 08:34:13.402724 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.402694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.404904 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.404874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"]
Sep 19 08:34:13.405231 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.405210    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"default-dockercfg-7wf8m\""
Sep 19 08:34:13.405440 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.405399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"cpu-accel-preset-kserve-self-signed-certs\""
Sep 19 08:34:13.405440 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.405429    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"seaweedfs-s3-creds\""
Sep 19 08:34:13.405628 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.405446    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"openshift-service-ca.crt\""
Sep 19 08:34:13.405628 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.405572    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"kube-root-ca.crt\""
Sep 19 08:34:13.553462 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.553422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-tmp-dir\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.553462 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.553476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-dshm\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.553850 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.553568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-kserve-provision-location\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.553850 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.553594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlpz\" (UniqueName: \"kubernetes.io/projected/75721484-5141-460b-b863-f7effa782ef3-kube-api-access-qnlpz\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.553850 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.553613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-model-cache\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.553850 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.553633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75721484-5141-460b-b863-f7effa782ef3-tls-certs\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.553850 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.553737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-home\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655069 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.654981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-kserve-provision-location\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655069 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlpz\" (UniqueName: \"kubernetes.io/projected/75721484-5141-460b-b863-f7effa782ef3-kube-api-access-qnlpz\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655069 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-model-cache\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655310 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75721484-5141-460b-b863-f7effa782ef3-tls-certs\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655403 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-home\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655497 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-tmp-dir\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655551 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-dshm\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655605 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-kserve-provision-location\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655662 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-model-cache\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655802 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-home\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.655909 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.655887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-tmp-dir\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.657675 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.657659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-dshm\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.657855 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.657839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75721484-5141-460b-b863-f7effa782ef3-tls-certs\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.663141 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.663119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlpz\" (UniqueName: \"kubernetes.io/projected/75721484-5141-460b-b863-f7effa782ef3-kube-api-access-qnlpz\") pod \"cpu-accel-preset-kserve-7b55796b88-xjrt5\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:13.715047 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:13.715005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:34:14.008861 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:14.008833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"]
Sep 19 08:34:14.010863 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:34:14.010826    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75721484_5141_460b_b863_f7effa782ef3.slice/crio-da571d380dc68a0f3556a31efa990eb3f607b05af076055bfbc136a924bb6dab WatchSource:0}: Error finding container da571d380dc68a0f3556a31efa990eb3f607b05af076055bfbc136a924bb6dab: Status 404 returned error can't find the container with id da571d380dc68a0f3556a31efa990eb3f607b05af076055bfbc136a924bb6dab
Sep 19 08:34:14.304069 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:14.303983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5" event={"ID":"75721484-5141-460b-b863-f7effa782ef3","Type":"ContainerStarted","Data":"a508653980c2aeaab34a145ef4d6e72e48b59fbc93b996555f33067cac2a63e8"}
Sep 19 08:34:14.304069 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:14.304033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5" event={"ID":"75721484-5141-460b-b863-f7effa782ef3","Type":"ContainerStarted","Data":"da571d380dc68a0f3556a31efa990eb3f607b05af076055bfbc136a924bb6dab"}
Sep 19 08:34:18.339394 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:18.339333    2577 generic.go:356] "Generic (PLEG): container finished" podID="75721484-5141-460b-b863-f7effa782ef3" containerID="a508653980c2aeaab34a145ef4d6e72e48b59fbc93b996555f33067cac2a63e8" exitCode=0
Sep 19 08:34:18.339984 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:18.339411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5" event={"ID":"75721484-5141-460b-b863-f7effa782ef3","Type":"ContainerDied","Data":"a508653980c2aeaab34a145ef4d6e72e48b59fbc93b996555f33067cac2a63e8"}
Sep 19 08:34:20.063316 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.062734    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"]
Sep 19 08:34:20.067598 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.067261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:20.072022 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.071207    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"]
Sep 19 08:34:20.072022 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.071802    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 19 08:34:20.073139 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.072856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-86kf7\""
Sep 19 08:34:20.073139 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.072984    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 19 08:34:20.073139 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.073019    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-tms8v\""
Sep 19 08:34:20.073406 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.073269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 19 08:34:20.105881 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.105854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18f9005-8165-4b6e-bf8a-0ba153f3c254-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd\" (UID: \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:20.106067 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.106044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6vg\" (UniqueName: \"kubernetes.io/projected/b18f9005-8165-4b6e-bf8a-0ba153f3c254-kube-api-access-9f6vg\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd\" (UID: \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:20.207024 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.206496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18f9005-8165-4b6e-bf8a-0ba153f3c254-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd\" (UID: \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:20.207024 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.206614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6vg\" (UniqueName: \"kubernetes.io/projected/b18f9005-8165-4b6e-bf8a-0ba153f3c254-kube-api-access-9f6vg\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd\" (UID: \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:20.210420 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.210346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18f9005-8165-4b6e-bf8a-0ba153f3c254-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd\" (UID: \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:20.216238 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.216190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6vg\" (UniqueName: \"kubernetes.io/projected/b18f9005-8165-4b6e-bf8a-0ba153f3c254-kube-api-access-9f6vg\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd\" (UID: \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:20.384150 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.383950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:20.678543 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:20.678512    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"]
Sep 19 08:34:20.680650 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:34:20.680620    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18f9005_8165_4b6e_bf8a_0ba153f3c254.slice/crio-0ffe5684d563b0250884eb5b07ded54395e517e4dc97c34e9302fa6d6ebe643f WatchSource:0}: Error finding container 0ffe5684d563b0250884eb5b07ded54395e517e4dc97c34e9302fa6d6ebe643f: Status 404 returned error can't find the container with id 0ffe5684d563b0250884eb5b07ded54395e517e4dc97c34e9302fa6d6ebe643f
Sep 19 08:34:21.368038 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:21.368000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" event={"ID":"b18f9005-8165-4b6e-bf8a-0ba153f3c254","Type":"ContainerStarted","Data":"0ffe5684d563b0250884eb5b07ded54395e517e4dc97c34e9302fa6d6ebe643f"}
Sep 19 08:34:22.573248 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:22.573208    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"]
Sep 19 08:34:25.430278 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:25.430181    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" event={"ID":"b18f9005-8165-4b6e-bf8a-0ba153f3c254","Type":"ContainerStarted","Data":"174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301"}
Sep 19 08:34:25.431053 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:25.431021    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:25.451125 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:25.451066    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" podStartSLOduration=0.952574926 podStartE2EDuration="5.451046499s" totalImagesPullingTime="4.498471573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:25.450487942 +0000 UTC m=+833.132071375" watchObservedRunningTime="2026-09-19 08:34:25.451046499 +0000 UTC m=+833.132629935"
Sep 19 08:34:30.474680 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:30.474571    2577 generic.go:356] "Generic (PLEG): container finished" podID="f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67" containerID="51e7e6b0a015ca01432c5a67b51bb186ab5c1e8b3109943b3e679f793c414c5a" exitCode=0
Sep 19 08:34:30.474680 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:30.474646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" event={"ID":"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67","Type":"ContainerDied","Data":"51e7e6b0a015ca01432c5a67b51bb186ab5c1e8b3109943b3e679f793c414c5a"}
Sep 19 08:34:30.475232 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:30.475088    2577 scope.go:117] "RemoveContainer" containerID="51e7e6b0a015ca01432c5a67b51bb186ab5c1e8b3109943b3e679f793c414c5a"
Sep 19 08:34:30.476331 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:30.476293    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:34:30.476455 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:30.476434    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:34:30.476519 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:30.476490    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:34:31.483938 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:31.483892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" event={"ID":"f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67","Type":"ContainerStarted","Data":"82918838e9df6a1bcb49b7a2ebfa542bb42390a2ccee2508352906bf5e612019"}
Sep 19 08:34:31.644077 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:31.644033    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:34:31.645565 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:31.645522    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 19 08:34:31.645719 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:31.645586    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" podUID="f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 19 08:34:32.559531 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:32.559496    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:34:32.644045 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:32.643958    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 19 08:34:32.644045 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:32.644037    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" podUID="f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 19 08:34:33.645009 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:33.644924    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 19 08:34:33.645009 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:33.644996    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" podUID="f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 19 08:34:34.644823 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:34.644699    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 19 08:34:34.644823 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:34.644765    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" podUID="f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 19 08:34:35.644478 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:35.644439    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 19 08:34:35.644937 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:35.644507    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" podUID="f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 19 08:34:36.644533 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:36.644492    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 19 08:34:36.644897 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:36.644564    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl" podUID="f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 19 08:34:37.649737 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:37.649701    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:34:37.651032 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:37.651007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl"
Sep 19 08:34:57.447186 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:57.447155    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:34:59.424181 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:34:59.424148    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"]
Sep 19 08:35:07.447241 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:07.447197    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" podUID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:35:09.765522 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:09.765479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5" event={"ID":"75721484-5141-460b-b863-f7effa782ef3","Type":"ContainerStarted","Data":"a9059bcb00eeb69ac4f9706e09588c86176a00f39cdcd4bcf9866dca733cb929"}
Sep 19 08:35:09.789409 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:09.789336    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5" podStartSLOduration=2.547160833 podStartE2EDuration="56.789317922s" totalImagesPullingTime="50.688697708s" totalInitContainerRuntime="3.553459381s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:35:09.786446302 +0000 UTC m=+877.468029738" watchObservedRunningTime="2026-09-19 08:35:09.789317922 +0000 UTC m=+877.470901354"
Sep 19 08:35:10.715094 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:10.715059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:35:14.448943 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:14.448902    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" podUID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" containerName="main" containerID="cri-o://174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301" gracePeriod=45
Sep 19 08:35:17.446482 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:17.446400    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" podUID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:35:24.785043 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:24.785002    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5" podUID="75721484-5141-460b-b863-f7effa782ef3" containerName="main" containerID="cri-o://a9059bcb00eeb69ac4f9706e09588c86176a00f39cdcd4bcf9866dca733cb929" gracePeriod=45
Sep 19 08:35:27.446436 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:27.446381    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" podUID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:35:27.446870 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:27.446542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:35:34.482870 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:34.482839    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:35:34.486610 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:34.486590    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:35:37.446964 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:37.446915    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" podUID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:35:44.695135 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.695107    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:35:44.847400 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.847309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18f9005-8165-4b6e-bf8a-0ba153f3c254-tls-certs\") pod \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\" (UID: \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\") "
Sep 19 08:35:44.847542 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.847417    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f6vg\" (UniqueName: \"kubernetes.io/projected/b18f9005-8165-4b6e-bf8a-0ba153f3c254-kube-api-access-9f6vg\") pod \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\" (UID: \"b18f9005-8165-4b6e-bf8a-0ba153f3c254\") "
Sep 19 08:35:44.849611 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.849576    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18f9005-8165-4b6e-bf8a-0ba153f3c254-kube-api-access-9f6vg" (OuterVolumeSpecName: "kube-api-access-9f6vg") pod "b18f9005-8165-4b6e-bf8a-0ba153f3c254" (UID: "b18f9005-8165-4b6e-bf8a-0ba153f3c254"). InnerVolumeSpecName "kube-api-access-9f6vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:35:44.849746 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.849713    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18f9005-8165-4b6e-bf8a-0ba153f3c254-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b18f9005-8165-4b6e-bf8a-0ba153f3c254" (UID: "b18f9005-8165-4b6e-bf8a-0ba153f3c254"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:35:44.948351 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.948315    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6vg\" (UniqueName: \"kubernetes.io/projected/b18f9005-8165-4b6e-bf8a-0ba153f3c254-kube-api-access-9f6vg\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:35:44.948351 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.948344    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b18f9005-8165-4b6e-bf8a-0ba153f3c254-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:35:44.966240 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.966207    2577 generic.go:356] "Generic (PLEG): container finished" podID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" containerID="174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301" exitCode=0
Sep 19 08:35:44.966383 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.966251    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"
Sep 19 08:35:44.966383 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.966265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" event={"ID":"b18f9005-8165-4b6e-bf8a-0ba153f3c254","Type":"ContainerDied","Data":"174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301"}
Sep 19 08:35:44.966383 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.966308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd" event={"ID":"b18f9005-8165-4b6e-bf8a-0ba153f3c254","Type":"ContainerDied","Data":"0ffe5684d563b0250884eb5b07ded54395e517e4dc97c34e9302fa6d6ebe643f"}
Sep 19 08:35:44.966383 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.966332    2577 scope.go:117] "RemoveContainer" containerID="174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301"
Sep 19 08:35:44.977766 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.977745    2577 scope.go:117] "RemoveContainer" containerID="174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301"
Sep 19 08:35:44.978107 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:35:44.978088    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301\": container with ID starting with 174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301 not found: ID does not exist" containerID="174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301"
Sep 19 08:35:44.978161 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.978117    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301"} err="failed to get container status \"174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301\": rpc error: code = NotFound desc = could not find container \"174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301\": container with ID starting with 174f29f2176c3595ea9d8a8b70bc98ea0764c282835379775d18d6c0505e5301 not found: ID does not exist"
Sep 19 08:35:44.989316 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.989289    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"]
Sep 19 08:35:44.994913 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:44.994889    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schenlrqd"]
Sep 19 08:35:46.505398 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:46.505351    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" path="/var/lib/kubelet/pods/b18f9005-8165-4b6e-bf8a-0ba153f3c254/volumes"
Sep 19 08:35:55.120886 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.120851    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"]
Sep 19 08:35:55.121510 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.121224    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" containerName="main"
Sep 19 08:35:55.121510 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.121235    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" containerName="main"
Sep 19 08:35:55.121510 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.121299    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18f9005-8165-4b6e-bf8a-0ba153f3c254" containerName="main"
Sep 19 08:35:55.124412 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.124387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:35:55.128786 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.128757    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 19 08:35:55.128952 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.128765    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-w77nd\""
Sep 19 08:35:55.130544 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.130041    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 19 08:35:55.130544 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.130177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-bqpjp\""
Sep 19 08:35:55.130544 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.130262    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 19 08:35:55.134915 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.134053    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"]
Sep 19 08:35:55.141158 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.141130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ls7x\" (UniqueName: \"kubernetes.io/projected/88fcdadf-d99a-4880-92c7-09a80a3e01d9-kube-api-access-6ls7x\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86\" (UID: \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:35:55.141254 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.141204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88fcdadf-d99a-4880-92c7-09a80a3e01d9-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86\" (UID: \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:35:55.241818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.241783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ls7x\" (UniqueName: \"kubernetes.io/projected/88fcdadf-d99a-4880-92c7-09a80a3e01d9-kube-api-access-6ls7x\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86\" (UID: \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:35:55.242000 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.241848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88fcdadf-d99a-4880-92c7-09a80a3e01d9-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86\" (UID: \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:35:55.252042 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.252012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ls7x\" (UniqueName: \"kubernetes.io/projected/88fcdadf-d99a-4880-92c7-09a80a3e01d9-kube-api-access-6ls7x\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86\" (UID: \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:35:55.252203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.252172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88fcdadf-d99a-4880-92c7-09a80a3e01d9-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86\" (UID: \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:35:55.439083 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.439012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:35:55.721661 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:55.721552    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"]
Sep 19 08:35:55.726188 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:35:55.726143    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fcdadf_d99a_4880_92c7_09a80a3e01d9.slice/crio-e87950e7d7ce282ebf4697a1cd4b43ebcf04ace8ad5598252e94b4f20862e523 WatchSource:0}: Error finding container e87950e7d7ce282ebf4697a1cd4b43ebcf04ace8ad5598252e94b4f20862e523: Status 404 returned error can't find the container with id e87950e7d7ce282ebf4697a1cd4b43ebcf04ace8ad5598252e94b4f20862e523
Sep 19 08:35:56.037706 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:56.037600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" event={"ID":"88fcdadf-d99a-4880-92c7-09a80a3e01d9","Type":"ContainerStarted","Data":"95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f"}
Sep 19 08:35:56.037706 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:56.037668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" event={"ID":"88fcdadf-d99a-4880-92c7-09a80a3e01d9","Type":"ContainerStarted","Data":"e87950e7d7ce282ebf4697a1cd4b43ebcf04ace8ad5598252e94b4f20862e523"}
Sep 19 08:35:56.037917 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:56.037791    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:35:56.054449 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:35:56.054248    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" podStartSLOduration=1.054228965 podStartE2EDuration="1.054228965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:35:56.053612138 +0000 UTC m=+923.735195573" watchObservedRunningTime="2026-09-19 08:35:56.054228965 +0000 UTC m=+923.735812398"
Sep 19 08:36:10.121587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.121556    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-7b55796b88-xjrt5_75721484-5141-460b-b863-f7effa782ef3/main/0.log"
Sep 19 08:36:10.122036 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.121937    2577 generic.go:356] "Generic (PLEG): container finished" podID="75721484-5141-460b-b863-f7effa782ef3" containerID="a9059bcb00eeb69ac4f9706e09588c86176a00f39cdcd4bcf9866dca733cb929" exitCode=137
Sep 19 08:36:10.122095 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.122025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5" event={"ID":"75721484-5141-460b-b863-f7effa782ef3","Type":"ContainerDied","Data":"a9059bcb00eeb69ac4f9706e09588c86176a00f39cdcd4bcf9866dca733cb929"}
Sep 19 08:36:10.433102 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.433075    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-7b55796b88-xjrt5_75721484-5141-460b-b863-f7effa782ef3/main/0.log"
Sep 19 08:36:10.435846 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.435822    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:36:10.482774 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.482738    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-kserve-provision-location\") pod \"75721484-5141-460b-b863-f7effa782ef3\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") "
Sep 19 08:36:10.482949 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.482788    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-model-cache\") pod \"75721484-5141-460b-b863-f7effa782ef3\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") "
Sep 19 08:36:10.482949 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.482833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-tmp-dir\") pod \"75721484-5141-460b-b863-f7effa782ef3\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") "
Sep 19 08:36:10.482949 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.482864    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnlpz\" (UniqueName: \"kubernetes.io/projected/75721484-5141-460b-b863-f7effa782ef3-kube-api-access-qnlpz\") pod \"75721484-5141-460b-b863-f7effa782ef3\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") "
Sep 19 08:36:10.483088 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.482975    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-dshm\") pod \"75721484-5141-460b-b863-f7effa782ef3\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") "
Sep 19 08:36:10.483088 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.483063    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-model-cache" (OuterVolumeSpecName: "model-cache") pod "75721484-5141-460b-b863-f7effa782ef3" (UID: "75721484-5141-460b-b863-f7effa782ef3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:10.483167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.483126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75721484-5141-460b-b863-f7effa782ef3-tls-certs\") pod \"75721484-5141-460b-b863-f7effa782ef3\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") "
Sep 19 08:36:10.483167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.483157    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-home\") pod \"75721484-5141-460b-b863-f7effa782ef3\" (UID: \"75721484-5141-460b-b863-f7effa782ef3\") "
Sep 19 08:36:10.483504 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.483468    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:10.483835 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.483806    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-home" (OuterVolumeSpecName: "home") pod "75721484-5141-460b-b863-f7effa782ef3" (UID: "75721484-5141-460b-b863-f7effa782ef3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:10.484056 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.484025    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75721484-5141-460b-b863-f7effa782ef3" (UID: "75721484-5141-460b-b863-f7effa782ef3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:10.485622 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.485588    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75721484-5141-460b-b863-f7effa782ef3-kube-api-access-qnlpz" (OuterVolumeSpecName: "kube-api-access-qnlpz") pod "75721484-5141-460b-b863-f7effa782ef3" (UID: "75721484-5141-460b-b863-f7effa782ef3"). InnerVolumeSpecName "kube-api-access-qnlpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:36:10.485622 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.485592    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75721484-5141-460b-b863-f7effa782ef3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75721484-5141-460b-b863-f7effa782ef3" (UID: "75721484-5141-460b-b863-f7effa782ef3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:36:10.485802 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.485630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-dshm" (OuterVolumeSpecName: "dshm") pod "75721484-5141-460b-b863-f7effa782ef3" (UID: "75721484-5141-460b-b863-f7effa782ef3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:10.542931 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.542877    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "75721484-5141-460b-b863-f7effa782ef3" (UID: "75721484-5141-460b-b863-f7effa782ef3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:10.584898 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.584820    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:10.584898 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.584858    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75721484-5141-460b-b863-f7effa782ef3-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:10.584898 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.584897    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:10.584898 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.584907    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:10.585172 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.584915    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75721484-5141-460b-b863-f7effa782ef3-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:10.585172 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:10.584930    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnlpz\" (UniqueName: \"kubernetes.io/projected/75721484-5141-460b-b863-f7effa782ef3-kube-api-access-qnlpz\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:11.128862 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:11.128829    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-7b55796b88-xjrt5_75721484-5141-460b-b863-f7effa782ef3/main/0.log"
Sep 19 08:36:11.129329 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:11.129311    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"
Sep 19 08:36:11.129390 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:11.129319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5" event={"ID":"75721484-5141-460b-b863-f7effa782ef3","Type":"ContainerDied","Data":"da571d380dc68a0f3556a31efa990eb3f607b05af076055bfbc136a924bb6dab"}
Sep 19 08:36:11.129436 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:11.129409    2577 scope.go:117] "RemoveContainer" containerID="a9059bcb00eeb69ac4f9706e09588c86176a00f39cdcd4bcf9866dca733cb929"
Sep 19 08:36:11.139003 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:11.138983    2577 scope.go:117] "RemoveContainer" containerID="a508653980c2aeaab34a145ef4d6e72e48b59fbc93b996555f33067cac2a63e8"
Sep 19 08:36:11.152529 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:11.152499    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"]
Sep 19 08:36:11.158441 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:11.158411    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-xjrt5"]
Sep 19 08:36:12.503650 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:12.503619    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75721484-5141-460b-b863-f7effa782ef3" path="/var/lib/kubelet/pods/75721484-5141-460b-b863-f7effa782ef3/volumes"
Sep 19 08:36:16.767937 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.767901    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"]
Sep 19 08:36:16.768483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.768304    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75721484-5141-460b-b863-f7effa782ef3" containerName="main"
Sep 19 08:36:16.768483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.768315    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75721484-5141-460b-b863-f7effa782ef3" containerName="main"
Sep 19 08:36:16.768483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.768329    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75721484-5141-460b-b863-f7effa782ef3" containerName="storage-initializer"
Sep 19 08:36:16.768483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.768335    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75721484-5141-460b-b863-f7effa782ef3" containerName="storage-initializer"
Sep 19 08:36:16.768483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.768410    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="75721484-5141-460b-b863-f7effa782ef3" containerName="main"
Sep 19 08:36:16.773224 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.773190    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.775952 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.775924    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"openshift-service-ca.crt\""
Sep 19 08:36:16.776082 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.775939    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"seaweedfs-s3-creds\""
Sep 19 08:36:16.776179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.776017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"cpu-accel-precedence-kserve-self-signed-certs\""
Sep 19 08:36:16.776179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.776059    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"default-dockercfg-5vsg6\""
Sep 19 08:36:16.776342 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.776184    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"kube-root-ca.crt\""
Sep 19 08:36:16.783479 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.783442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"]
Sep 19 08:36:16.841189 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.841149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tmp-dir\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.841414 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.841207    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.841414 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.841252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xglxl\" (UniqueName: \"kubernetes.io/projected/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kube-api-access-xglxl\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.841414 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.841304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-home\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.841414 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.841337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tls-certs\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.841631 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.841481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-dshm\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.841631 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.841533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-model-cache\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942160 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-dshm\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942160 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-model-cache\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942429 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tmp-dir\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942429 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942550 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xglxl\" (UniqueName: \"kubernetes.io/projected/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kube-api-access-xglxl\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942550 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-home\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942657 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tls-certs\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942735 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-model-cache\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942794 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942794 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tmp-dir\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.942895 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.942875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-home\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.944500 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.944474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-dshm\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.944809 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.944792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tls-certs\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:16.951072 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:16.951052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xglxl\" (UniqueName: \"kubernetes.io/projected/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kube-api-access-xglxl\") pod \"cpu-accel-precedence-kserve-5477c794d4-4kcnd\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:17.085267 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:17.085177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:17.364116 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:36:17.364089    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c14f6ef_1d99_4b74_9498_ae38713db1fc.slice/crio-e6c373d8d6a9e8c41c9e7d0dfba39c838501664e32cb65e4ca30094bc0fc9c95 WatchSource:0}: Error finding container e6c373d8d6a9e8c41c9e7d0dfba39c838501664e32cb65e4ca30094bc0fc9c95: Status 404 returned error can't find the container with id e6c373d8d6a9e8c41c9e7d0dfba39c838501664e32cb65e4ca30094bc0fc9c95
Sep 19 08:36:17.364283 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:17.364256    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"]
Sep 19 08:36:18.171966 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:18.171925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd" event={"ID":"9c14f6ef-1d99-4b74-9498-ae38713db1fc","Type":"ContainerStarted","Data":"00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551"}
Sep 19 08:36:18.172462 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:18.171980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd" event={"ID":"9c14f6ef-1d99-4b74-9498-ae38713db1fc","Type":"ContainerStarted","Data":"e6c373d8d6a9e8c41c9e7d0dfba39c838501664e32cb65e4ca30094bc0fc9c95"}
Sep 19 08:36:21.458511 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.457658    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"]
Sep 19 08:36:21.458511 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.457937    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd" podUID="9c14f6ef-1d99-4b74-9498-ae38713db1fc" containerName="storage-initializer" containerID="cri-o://00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551" gracePeriod=60
Sep 19 08:36:21.603048 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.603021    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:21.685597 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685513    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xglxl\" (UniqueName: \"kubernetes.io/projected/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kube-api-access-xglxl\") pod \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") "
Sep 19 08:36:21.685597 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685557    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-home\") pod \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") "
Sep 19 08:36:21.685597 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tmp-dir\") pod \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") "
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685635    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-model-cache\") pod \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") "
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685664    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kserve-provision-location\") pod \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") "
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685718    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tls-certs\") pod \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") "
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685750    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-dshm\") pod \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\" (UID: \"9c14f6ef-1d99-4b74-9498-ae38713db1fc\") "
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-home" (OuterVolumeSpecName: "home") pod "9c14f6ef-1d99-4b74-9498-ae38713db1fc" (UID: "9c14f6ef-1d99-4b74-9498-ae38713db1fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685914    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c14f6ef-1d99-4b74-9498-ae38713db1fc" (UID: "9c14f6ef-1d99-4b74-9498-ae38713db1fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.685930    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c14f6ef-1d99-4b74-9498-ae38713db1fc" (UID: "9c14f6ef-1d99-4b74-9498-ae38713db1fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.686038    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.686056    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:21.686087 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.686072    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:21.687899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.687875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-dshm" (OuterVolumeSpecName: "dshm") pod "9c14f6ef-1d99-4b74-9498-ae38713db1fc" (UID: "9c14f6ef-1d99-4b74-9498-ae38713db1fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:21.688067 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.688049    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c14f6ef-1d99-4b74-9498-ae38713db1fc" (UID: "9c14f6ef-1d99-4b74-9498-ae38713db1fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:36:21.688130 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.688101    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kube-api-access-xglxl" (OuterVolumeSpecName: "kube-api-access-xglxl") pod "9c14f6ef-1d99-4b74-9498-ae38713db1fc" (UID: "9c14f6ef-1d99-4b74-9498-ae38713db1fc"). InnerVolumeSpecName "kube-api-access-xglxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:36:21.760433 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.760390    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c14f6ef-1d99-4b74-9498-ae38713db1fc" (UID: "9c14f6ef-1d99-4b74-9498-ae38713db1fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:21.787488 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.787460    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c14f6ef-1d99-4b74-9498-ae38713db1fc-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:21.787488 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.787487    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:21.787671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.787496    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xglxl\" (UniqueName: \"kubernetes.io/projected/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kube-api-access-xglxl\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:21.787671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:21.787505    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c14f6ef-1d99-4b74-9498-ae38713db1fc-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:36:22.196343 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.196305    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c14f6ef-1d99-4b74-9498-ae38713db1fc" containerID="00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551" exitCode=0
Sep 19 08:36:22.196567 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.196356    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"
Sep 19 08:36:22.196567 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.196391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd" event={"ID":"9c14f6ef-1d99-4b74-9498-ae38713db1fc","Type":"ContainerDied","Data":"00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551"}
Sep 19 08:36:22.196567 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.196429    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd" event={"ID":"9c14f6ef-1d99-4b74-9498-ae38713db1fc","Type":"ContainerDied","Data":"e6c373d8d6a9e8c41c9e7d0dfba39c838501664e32cb65e4ca30094bc0fc9c95"}
Sep 19 08:36:22.196567 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.196445    2577 scope.go:117] "RemoveContainer" containerID="00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551"
Sep 19 08:36:22.209395 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.209375    2577 scope.go:117] "RemoveContainer" containerID="00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551"
Sep 19 08:36:22.209724 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:36:22.209701    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551\": container with ID starting with 00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551 not found: ID does not exist" containerID="00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551"
Sep 19 08:36:22.209762 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.209734    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551"} err="failed to get container status \"00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551\": rpc error: code = NotFound desc = could not find container \"00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551\": container with ID starting with 00774d7a150a39da939ae2cda7fc93da6433b3f5f70548760cf4ea904d002551 not found: ID does not exist"
Sep 19 08:36:22.242185 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.242154    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"]
Sep 19 08:36:22.247459 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.247431    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-4kcnd"]
Sep 19 08:36:22.504462 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:22.504381    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c14f6ef-1d99-4b74-9498-ae38713db1fc" path="/var/lib/kubelet/pods/9c14f6ef-1d99-4b74-9498-ae38713db1fc/volumes"
Sep 19 08:36:27.045242 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:27.045208    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:36:36.589225 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.589191    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"]
Sep 19 08:36:36.589640 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.589625    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c14f6ef-1d99-4b74-9498-ae38713db1fc" containerName="storage-initializer"
Sep 19 08:36:36.589688 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.589642    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c14f6ef-1d99-4b74-9498-ae38713db1fc" containerName="storage-initializer"
Sep 19 08:36:36.589758 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.589708    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c14f6ef-1d99-4b74-9498-ae38713db1fc" containerName="storage-initializer"
Sep 19 08:36:36.594353 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.594330    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.597304 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.597281    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Sep 19 08:36:36.597449 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.597292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-nk589\""
Sep 19 08:36:36.597449 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.597313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"seaweedfs-s3-creds\""
Sep 19 08:36:36.597584 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.597510    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Sep 19 08:36:36.597676 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.597657    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:36:36.597864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.597838    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Sep 19 08:36:36.607922 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.607894    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"]
Sep 19 08:36:36.714090 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.714054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.714284 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.714117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4c137-22dc-4b19-8920-280f98a0f067-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.714284 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.714139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.714284 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.714176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.714284 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.714204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.714284 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.714222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba4c137-22dc-4b19-8920-280f98a0f067-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.714284 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.714246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.714284 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.714275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6825f\" (UniqueName: \"kubernetes.io/projected/6ba4c137-22dc-4b19-8920-280f98a0f067-kube-api-access-6825f\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815493 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4c137-22dc-4b19-8920-280f98a0f067-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815493 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815493 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815493 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815701 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba4c137-22dc-4b19-8920-280f98a0f067-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815701 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815701 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6825f\" (UniqueName: \"kubernetes.io/projected/6ba4c137-22dc-4b19-8920-280f98a0f067-kube-api-access-6825f\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815863 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815919 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.815919 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.816033 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.815955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.816214 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.816195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4c137-22dc-4b19-8920-280f98a0f067-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.817804 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.817774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.818179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.818158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba4c137-22dc-4b19-8920-280f98a0f067-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.825798 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.825771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6825f\" (UniqueName: \"kubernetes.io/projected/6ba4c137-22dc-4b19-8920-280f98a0f067-kube-api-access-6825f\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:36.904168 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:36.904128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:37.183240 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:37.183211    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"]
Sep 19 08:36:37.185034 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:36:37.185006    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba4c137_22dc_4b19_8920_280f98a0f067.slice/crio-13d669d14ccf00824a31e4490049761dcd7653a27137b3e36c4e54330607aa77 WatchSource:0}: Error finding container 13d669d14ccf00824a31e4490049761dcd7653a27137b3e36c4e54330607aa77: Status 404 returned error can't find the container with id 13d669d14ccf00824a31e4490049761dcd7653a27137b3e36c4e54330607aa77
Sep 19 08:36:37.281760 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:37.281723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" event={"ID":"6ba4c137-22dc-4b19-8920-280f98a0f067","Type":"ContainerStarted","Data":"799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c"}
Sep 19 08:36:37.281952 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:37.281771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" event={"ID":"6ba4c137-22dc-4b19-8920-280f98a0f067","Type":"ContainerStarted","Data":"13d669d14ccf00824a31e4490049761dcd7653a27137b3e36c4e54330607aa77"}
Sep 19 08:36:40.299750 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:40.299713    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerID="799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c" exitCode=0
Sep 19 08:36:40.300177 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:40.299787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" event={"ID":"6ba4c137-22dc-4b19-8920-280f98a0f067","Type":"ContainerDied","Data":"799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c"}
Sep 19 08:36:41.307939 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:41.307900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" event={"ID":"6ba4c137-22dc-4b19-8920-280f98a0f067","Type":"ContainerStarted","Data":"1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f"}
Sep 19 08:36:41.331107 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:41.331056    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podStartSLOduration=3.043220903 podStartE2EDuration="5.331037689s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.287816786s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:36:41.328190573 +0000 UTC m=+969.009774006" watchObservedRunningTime="2026-09-19 08:36:41.331037689 +0000 UTC m=+969.012621122"
Sep 19 08:36:41.904312 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:41.904265    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:46.904644 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:46.904553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:36:46.906485 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:46.906446    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 19 08:36:56.905165 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:56.905124    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 19 08:36:59.705286 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:36:59.705249    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"]
Sep 19 08:37:06.904547 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:06.904498    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 19 08:37:07.044803 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:07.044761    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" podUID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:37:14.728220 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:14.728174    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" podUID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" containerName="main" containerID="cri-o://95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f" gracePeriod=45
Sep 19 08:37:16.904963 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:16.904913    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 19 08:37:17.045077 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:17.045036    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" podUID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:37:26.905326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:26.905279    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 19 08:37:27.044637 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:27.044599    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" podUID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:37:27.044808 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:27.044702    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:37:36.904753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:36.904707    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 19 08:37:37.045297 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:37.045257    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" podUID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:37:44.980066 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:44.980029    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:37:45.020987 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.020899    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ls7x\" (UniqueName: \"kubernetes.io/projected/88fcdadf-d99a-4880-92c7-09a80a3e01d9-kube-api-access-6ls7x\") pod \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\" (UID: \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\") "
Sep 19 08:37:45.021150 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.020999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88fcdadf-d99a-4880-92c7-09a80a3e01d9-tls-certs\") pod \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\" (UID: \"88fcdadf-d99a-4880-92c7-09a80a3e01d9\") "
Sep 19 08:37:45.023231 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.023198    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fcdadf-d99a-4880-92c7-09a80a3e01d9-kube-api-access-6ls7x" (OuterVolumeSpecName: "kube-api-access-6ls7x") pod "88fcdadf-d99a-4880-92c7-09a80a3e01d9" (UID: "88fcdadf-d99a-4880-92c7-09a80a3e01d9"). InnerVolumeSpecName "kube-api-access-6ls7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:37:45.023231 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.023212    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88fcdadf-d99a-4880-92c7-09a80a3e01d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88fcdadf-d99a-4880-92c7-09a80a3e01d9" (UID: "88fcdadf-d99a-4880-92c7-09a80a3e01d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:37:45.121896 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.121854    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ls7x\" (UniqueName: \"kubernetes.io/projected/88fcdadf-d99a-4880-92c7-09a80a3e01d9-kube-api-access-6ls7x\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:37:45.121896 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.121889    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88fcdadf-d99a-4880-92c7-09a80a3e01d9-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:37:45.688140 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.688103    2577 generic.go:356] "Generic (PLEG): container finished" podID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" containerID="95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f" exitCode=0
Sep 19 08:37:45.688325 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.688148    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"
Sep 19 08:37:45.688325 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.688186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" event={"ID":"88fcdadf-d99a-4880-92c7-09a80a3e01d9","Type":"ContainerDied","Data":"95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f"}
Sep 19 08:37:45.688325 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.688242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86" event={"ID":"88fcdadf-d99a-4880-92c7-09a80a3e01d9","Type":"ContainerDied","Data":"e87950e7d7ce282ebf4697a1cd4b43ebcf04ace8ad5598252e94b4f20862e523"}
Sep 19 08:37:45.688325 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.688265    2577 scope.go:117] "RemoveContainer" containerID="95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f"
Sep 19 08:37:45.699449 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.699431    2577 scope.go:117] "RemoveContainer" containerID="95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f"
Sep 19 08:37:45.699742 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:37:45.699725    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f\": container with ID starting with 95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f not found: ID does not exist" containerID="95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f"
Sep 19 08:37:45.699805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.699750    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f"} err="failed to get container status \"95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f\": rpc error: code = NotFound desc = could not find container \"95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f\": container with ID starting with 95b0661a364f23004e73ea3af0b97fc86da8d7e77d079bc6d40138aeb859211f not found: ID does not exist"
Sep 19 08:37:45.710546 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.710519    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"]
Sep 19 08:37:45.715584 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:45.715558    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97vw86"]
Sep 19 08:37:46.505387 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:46.505332    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" path="/var/lib/kubelet/pods/88fcdadf-d99a-4880-92c7-09a80a3e01d9/volumes"
Sep 19 08:37:46.905001 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:46.904953    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 19 08:37:56.919215 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:56.919186    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:37:56.926212 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:37:56.926194    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:38:02.875526 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.875488    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"]
Sep 19 08:38:02.875988 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.875887    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" containerName="main"
Sep 19 08:38:02.875988 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.875898    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" containerName="main"
Sep 19 08:38:02.875988 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.875959    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="88fcdadf-d99a-4880-92c7-09a80a3e01d9" containerName="main"
Sep 19 08:38:02.881996 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.881976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:02.884871 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.884847    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 19 08:38:02.885973 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.885948    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-cwklf\""
Sep 19 08:38:02.886093 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.886043    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 19 08:38:02.886159 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.886145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 19 08:38:02.888861 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.888839    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"]
Sep 19 08:38:02.979359 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.979326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:02.979359 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.979378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a895edf3-4891-4a7f-95c1-91af7e48af07-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:02.979584 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.979451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:02.979584 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.979476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s2kc\" (UniqueName: \"kubernetes.io/projected/a895edf3-4891-4a7f-95c1-91af7e48af07-kube-api-access-6s2kc\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:02.979584 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.979507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:02.979584 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:02.979556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.078547 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.078506    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"]
Sep 19 08:38:03.080447 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.080419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.080594 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.080455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a895edf3-4891-4a7f-95c1-91af7e48af07-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.080594 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.080475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.080594 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.080494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s2kc\" (UniqueName: \"kubernetes.io/projected/a895edf3-4891-4a7f-95c1-91af7e48af07-kube-api-access-6s2kc\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.080594 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.080534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.080594 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.080579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.080894 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.080870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.080952 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.080892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.080952 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.080941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.082851 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.082830    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.082964 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.082948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a895edf3-4891-4a7f-95c1-91af7e48af07-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.083236 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.083222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:38:03.085719 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.085702    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-mn77q\""
Sep 19 08:38:03.093711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.093689    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"]
Sep 19 08:38:03.093938 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.093920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s2kc\" (UniqueName: \"kubernetes.io/projected/a895edf3-4891-4a7f-95c1-91af7e48af07-kube-api-access-6s2kc\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-76mtn\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.181748 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.181657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21699e97-ad3f-4cbf-86e8-fb0a9146631e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9\" (UID: \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:38:03.181748 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.181709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9vl\" (UniqueName: \"kubernetes.io/projected/21699e97-ad3f-4cbf-86e8-fb0a9146631e-kube-api-access-hh9vl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9\" (UID: \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:38:03.192595 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.192568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:03.283052 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.283021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21699e97-ad3f-4cbf-86e8-fb0a9146631e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9\" (UID: \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:38:03.283198 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.283076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9vl\" (UniqueName: \"kubernetes.io/projected/21699e97-ad3f-4cbf-86e8-fb0a9146631e-kube-api-access-hh9vl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9\" (UID: \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:38:03.285426 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.285406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21699e97-ad3f-4cbf-86e8-fb0a9146631e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9\" (UID: \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:38:03.290203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.290180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9vl\" (UniqueName: \"kubernetes.io/projected/21699e97-ad3f-4cbf-86e8-fb0a9146631e-kube-api-access-hh9vl\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9\" (UID: \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:38:03.415442 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.415405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:38:03.469572 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.469327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"]
Sep 19 08:38:03.476874 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.476850    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:38:03.695941 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.695908    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"]
Sep 19 08:38:03.696815 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:38:03.696783    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21699e97_ad3f_4cbf_86e8_fb0a9146631e.slice/crio-6863bb115f1a1e782aa9636f6ca6c283f05d012e90fb5c27d8aebe05d87bde96 WatchSource:0}: Error finding container 6863bb115f1a1e782aa9636f6ca6c283f05d012e90fb5c27d8aebe05d87bde96: Status 404 returned error can't find the container with id 6863bb115f1a1e782aa9636f6ca6c283f05d012e90fb5c27d8aebe05d87bde96
Sep 19 08:38:03.799733 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.799700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn" event={"ID":"a895edf3-4891-4a7f-95c1-91af7e48af07","Type":"ContainerStarted","Data":"5dd6203468edddd85d05e33d536d43b52a3ce73effa5f43501486dd249dd1732"}
Sep 19 08:38:03.801747 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.801717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" event={"ID":"21699e97-ad3f-4cbf-86e8-fb0a9146631e","Type":"ContainerStarted","Data":"a2e5f62b73aae543bfcca7ea79362c84e883e7b45863c0ce9fc51b9d01cb6a03"}
Sep 19 08:38:03.801888 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.801772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" event={"ID":"21699e97-ad3f-4cbf-86e8-fb0a9146631e","Type":"ContainerStarted","Data":"6863bb115f1a1e782aa9636f6ca6c283f05d012e90fb5c27d8aebe05d87bde96"}
Sep 19 08:38:03.801984 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:03.801963    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:38:05.817914 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:05.817878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn" event={"ID":"a895edf3-4891-4a7f-95c1-91af7e48af07","Type":"ContainerStarted","Data":"5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3"}
Sep 19 08:38:05.835175 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:05.835130    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn" podStartSLOduration=2.280132375 podStartE2EDuration="3.835114503s" totalImagesPullingTime="1.554982128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:38:05.833961344 +0000 UTC m=+1053.515544777" watchObservedRunningTime="2026-09-19 08:38:05.835114503 +0000 UTC m=+1053.516697934"
Sep 19 08:38:05.836883 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:05.836843    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" podStartSLOduration=2.836834818 podStartE2EDuration="2.836834818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:38:03.821756259 +0000 UTC m=+1051.503339684" watchObservedRunningTime="2026-09-19 08:38:05.836834818 +0000 UTC m=+1053.518418250"
Sep 19 08:38:06.193119 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:06.193087    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:13.193708 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:13.193664    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:13.205873 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:13.205846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:13.870821 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:13.870785    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:22.697198 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:22.697120    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"]
Sep 19 08:38:34.809743 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:34.809705    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:38:37.016651 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.016611    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"]
Sep 19 08:38:37.018563 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.018526    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:38:37.019895 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.019870    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"]
Sep 19 08:38:37.029754 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:38:37.029721    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:38:37.029754 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:38:37.029754 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:38:37.029754 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3" cmd=["/bin/sleep","15"]
Sep 19 08:38:37.030035 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.029780    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn" message="" err=<
Sep 19 08:38:37.030035 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:38:37.030035 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:38:37.030035 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:38:37.030035 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:38:37.029798    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:38:37.030035 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:38:37.030035 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:38:37.030035 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn" podUID="a895edf3-4891-4a7f-95c1-91af7e48af07" containerName="main" containerID="cri-o://5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3"
Sep 19 08:38:37.030035 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.029839    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn" podUID="a895edf3-4891-4a7f-95c1-91af7e48af07" containerName="main" containerID="cri-o://5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3" gracePeriod=60
Sep 19 08:38:37.382437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.382410    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:37.486968 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.486936    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-model-cache\") pod \"a895edf3-4891-4a7f-95c1-91af7e48af07\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") "
Sep 19 08:38:37.487131 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487013    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s2kc\" (UniqueName: \"kubernetes.io/projected/a895edf3-4891-4a7f-95c1-91af7e48af07-kube-api-access-6s2kc\") pod \"a895edf3-4891-4a7f-95c1-91af7e48af07\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") "
Sep 19 08:38:37.487131 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487038    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a895edf3-4891-4a7f-95c1-91af7e48af07-tls-certs\") pod \"a895edf3-4891-4a7f-95c1-91af7e48af07\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") "
Sep 19 08:38:37.487131 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487061    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-tmp-dir\") pod \"a895edf3-4891-4a7f-95c1-91af7e48af07\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") "
Sep 19 08:38:37.487131 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-dshm\") pod \"a895edf3-4891-4a7f-95c1-91af7e48af07\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") "
Sep 19 08:38:37.487319 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487137    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-home\") pod \"a895edf3-4891-4a7f-95c1-91af7e48af07\" (UID: \"a895edf3-4891-4a7f-95c1-91af7e48af07\") "
Sep 19 08:38:37.487319 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487198    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-model-cache" (OuterVolumeSpecName: "model-cache") pod "a895edf3-4891-4a7f-95c1-91af7e48af07" (UID: "a895edf3-4891-4a7f-95c1-91af7e48af07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:37.487456 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a895edf3-4891-4a7f-95c1-91af7e48af07" (UID: "a895edf3-4891-4a7f-95c1-91af7e48af07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:37.487518 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487453    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:37.487518 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487476    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:37.487518 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.487469    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-home" (OuterVolumeSpecName: "home") pod "a895edf3-4891-4a7f-95c1-91af7e48af07" (UID: "a895edf3-4891-4a7f-95c1-91af7e48af07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:37.489317 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.489290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a895edf3-4891-4a7f-95c1-91af7e48af07-kube-api-access-6s2kc" (OuterVolumeSpecName: "kube-api-access-6s2kc") pod "a895edf3-4891-4a7f-95c1-91af7e48af07" (UID: "a895edf3-4891-4a7f-95c1-91af7e48af07"). InnerVolumeSpecName "kube-api-access-6s2kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:38:37.489317 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.489305    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a895edf3-4891-4a7f-95c1-91af7e48af07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a895edf3-4891-4a7f-95c1-91af7e48af07" (UID: "a895edf3-4891-4a7f-95c1-91af7e48af07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:38:37.489487 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.489401    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-dshm" (OuterVolumeSpecName: "dshm") pod "a895edf3-4891-4a7f-95c1-91af7e48af07" (UID: "a895edf3-4891-4a7f-95c1-91af7e48af07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:37.588874 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.588799    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:37.588874 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.588834    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a895edf3-4891-4a7f-95c1-91af7e48af07-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:37.588874 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.588842    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s2kc\" (UniqueName: \"kubernetes.io/projected/a895edf3-4891-4a7f-95c1-91af7e48af07-kube-api-access-6s2kc\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:37.588874 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.588852    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a895edf3-4891-4a7f-95c1-91af7e48af07-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:37.719231 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.719166    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main" containerID="cri-o://1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f" gracePeriod=45
Sep 19 08:38:37.920259 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.920221    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 19 08:38:37.992878 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.992847    2577 generic.go:356] "Generic (PLEG): container finished" podID="a895edf3-4891-4a7f-95c1-91af7e48af07" containerID="5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3" exitCode=0
Sep 19 08:38:37.993058 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.992911    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"
Sep 19 08:38:37.993058 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.992931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn" event={"ID":"a895edf3-4891-4a7f-95c1-91af7e48af07","Type":"ContainerDied","Data":"5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3"}
Sep 19 08:38:37.993058 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.992985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn" event={"ID":"a895edf3-4891-4a7f-95c1-91af7e48af07","Type":"ContainerDied","Data":"5dd6203468edddd85d05e33d536d43b52a3ce73effa5f43501486dd249dd1732"}
Sep 19 08:38:37.993058 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:37.993014    2577 scope.go:117] "RemoveContainer" containerID="5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3"
Sep 19 08:38:38.002160 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.002140    2577 scope.go:117] "RemoveContainer" containerID="5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3"
Sep 19 08:38:38.002403 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:38:38.002387    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3\": container with ID starting with 5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3 not found: ID does not exist" containerID="5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3"
Sep 19 08:38:38.002467 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.002411    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3"} err="failed to get container status \"5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3\": rpc error: code = NotFound desc = could not find container \"5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3\": container with ID starting with 5b5925b3baed7000253b89118219dea393e1aa754555dd8c1396c06dde456ed3 not found: ID does not exist"
Sep 19 08:38:38.015069 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.015045    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"]
Sep 19 08:38:38.021068 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.021046    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-76mtn"]
Sep 19 08:38:38.497337 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.497312    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:38:38.503833 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.503808    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a895edf3-4891-4a7f-95c1-91af7e48af07" path="/var/lib/kubelet/pods/a895edf3-4891-4a7f-95c1-91af7e48af07/volumes"
Sep 19 08:38:38.598392 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4c137-22dc-4b19-8920-280f98a0f067-cabundle-cert\") pod \"6ba4c137-22dc-4b19-8920-280f98a0f067\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") "
Sep 19 08:38:38.598392 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-kserve-provision-location\") pod \"6ba4c137-22dc-4b19-8920-280f98a0f067\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") "
Sep 19 08:38:38.598392 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-dshm\") pod \"6ba4c137-22dc-4b19-8920-280f98a0f067\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") "
Sep 19 08:38:38.598663 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6825f\" (UniqueName: \"kubernetes.io/projected/6ba4c137-22dc-4b19-8920-280f98a0f067-kube-api-access-6825f\") pod \"6ba4c137-22dc-4b19-8920-280f98a0f067\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") "
Sep 19 08:38:38.598663 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598505    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba4c137-22dc-4b19-8920-280f98a0f067-tls-certs\") pod \"6ba4c137-22dc-4b19-8920-280f98a0f067\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") "
Sep 19 08:38:38.598663 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598534    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-tmp-dir\") pod \"6ba4c137-22dc-4b19-8920-280f98a0f067\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") "
Sep 19 08:38:38.598663 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-model-cache\") pod \"6ba4c137-22dc-4b19-8920-280f98a0f067\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") "
Sep 19 08:38:38.598663 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-home\") pod \"6ba4c137-22dc-4b19-8920-280f98a0f067\" (UID: \"6ba4c137-22dc-4b19-8920-280f98a0f067\") "
Sep 19 08:38:38.598924 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba4c137-22dc-4b19-8920-280f98a0f067-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ba4c137-22dc-4b19-8920-280f98a0f067" (UID: "6ba4c137-22dc-4b19-8920-280f98a0f067"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:38:38.599038 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.598960    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba4c137-22dc-4b19-8920-280f98a0f067-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:38.599352 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.599326    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-home" (OuterVolumeSpecName: "home") pod "6ba4c137-22dc-4b19-8920-280f98a0f067" (UID: "6ba4c137-22dc-4b19-8920-280f98a0f067"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:38.599618 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.599595    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ba4c137-22dc-4b19-8920-280f98a0f067" (UID: "6ba4c137-22dc-4b19-8920-280f98a0f067"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:38.600866 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.600832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba4c137-22dc-4b19-8920-280f98a0f067-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ba4c137-22dc-4b19-8920-280f98a0f067" (UID: "6ba4c137-22dc-4b19-8920-280f98a0f067"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:38:38.600973 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.600868    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba4c137-22dc-4b19-8920-280f98a0f067-kube-api-access-6825f" (OuterVolumeSpecName: "kube-api-access-6825f") pod "6ba4c137-22dc-4b19-8920-280f98a0f067" (UID: "6ba4c137-22dc-4b19-8920-280f98a0f067"). InnerVolumeSpecName "kube-api-access-6825f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:38:38.600973 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.600870    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-dshm" (OuterVolumeSpecName: "dshm") pod "6ba4c137-22dc-4b19-8920-280f98a0f067" (UID: "6ba4c137-22dc-4b19-8920-280f98a0f067"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:38.611028 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.610996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ba4c137-22dc-4b19-8920-280f98a0f067" (UID: "6ba4c137-22dc-4b19-8920-280f98a0f067"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:38.621913 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.621866    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ba4c137-22dc-4b19-8920-280f98a0f067" (UID: "6ba4c137-22dc-4b19-8920-280f98a0f067"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:38.699354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.699320    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba4c137-22dc-4b19-8920-280f98a0f067-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:38.699354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.699348    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:38.699354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.699357    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:38.699606 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.699388    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:38.699606 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.699407    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:38.699606 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.699415    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba4c137-22dc-4b19-8920-280f98a0f067-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:38.699606 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:38.699424    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6825f\" (UniqueName: \"kubernetes.io/projected/6ba4c137-22dc-4b19-8920-280f98a0f067-kube-api-access-6825f\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:38:39.005984 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.005898    2577 generic.go:356] "Generic (PLEG): container finished" podID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerID="1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f" exitCode=0
Sep 19 08:38:39.005984 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.005949    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"
Sep 19 08:38:39.006197 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.005975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" event={"ID":"6ba4c137-22dc-4b19-8920-280f98a0f067","Type":"ContainerDied","Data":"1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f"}
Sep 19 08:38:39.006197 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.006032    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk" event={"ID":"6ba4c137-22dc-4b19-8920-280f98a0f067","Type":"ContainerDied","Data":"13d669d14ccf00824a31e4490049761dcd7653a27137b3e36c4e54330607aa77"}
Sep 19 08:38:39.006197 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.006060    2577 scope.go:117] "RemoveContainer" containerID="1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f"
Sep 19 08:38:39.015498 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.015482    2577 scope.go:117] "RemoveContainer" containerID="799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c"
Sep 19 08:38:39.023648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.023632    2577 scope.go:117] "RemoveContainer" containerID="1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f"
Sep 19 08:38:39.023921 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:38:39.023903    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f\": container with ID starting with 1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f not found: ID does not exist" containerID="1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f"
Sep 19 08:38:39.023966 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.023930    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f"} err="failed to get container status \"1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f\": rpc error: code = NotFound desc = could not find container \"1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f\": container with ID starting with 1942bede92f468e740435f33db3a67552d81d3b593b5175f2ecb0ca7cfe8ea5f not found: ID does not exist"
Sep 19 08:38:39.023966 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.023947    2577 scope.go:117] "RemoveContainer" containerID="799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c"
Sep 19 08:38:39.024188 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:38:39.024157    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c\": container with ID starting with 799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c not found: ID does not exist" containerID="799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c"
Sep 19 08:38:39.024248 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.024197    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c"} err="failed to get container status \"799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c\": rpc error: code = NotFound desc = could not find container \"799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c\": container with ID starting with 799f24a78664cf36ab12cac1ae7cd53a8a79aafda59291c6c745994dc05b8a9c not found: ID does not exist"
Sep 19 08:38:39.028283 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.028259    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"]
Sep 19 08:38:39.031938 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:39.031918    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-qq4dk"]
Sep 19 08:38:40.503971 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:40.503938    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" path="/var/lib/kubelet/pods/6ba4c137-22dc-4b19-8920-280f98a0f067/volumes"
Sep 19 08:38:47.019214 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:47.019175    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:38:52.041519 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:52.041449    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main" containerID="cri-o://a2e5f62b73aae543bfcca7ea79362c84e883e7b45863c0ce9fc51b9d01cb6a03" gracePeriod=45
Sep 19 08:38:57.019173 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:38:57.019131    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:39:07.019115 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:07.019028    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:39:17.018359 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:17.018322    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:39:22.231631 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:22.231596    2577 generic.go:356] "Generic (PLEG): container finished" podID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerID="a2e5f62b73aae543bfcca7ea79362c84e883e7b45863c0ce9fc51b9d01cb6a03" exitCode=0
Sep 19 08:39:22.231984 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:22.231667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" event={"ID":"21699e97-ad3f-4cbf-86e8-fb0a9146631e","Type":"ContainerDied","Data":"a2e5f62b73aae543bfcca7ea79362c84e883e7b45863c0ce9fc51b9d01cb6a03"}
Sep 19 08:39:22.285666 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:22.285643    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:39:22.367400 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:22.367377    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21699e97-ad3f-4cbf-86e8-fb0a9146631e-tls-certs\") pod \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\" (UID: \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\") "
Sep 19 08:39:22.367557 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:22.367409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh9vl\" (UniqueName: \"kubernetes.io/projected/21699e97-ad3f-4cbf-86e8-fb0a9146631e-kube-api-access-hh9vl\") pod \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\" (UID: \"21699e97-ad3f-4cbf-86e8-fb0a9146631e\") "
Sep 19 08:39:22.369350 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:22.369324    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21699e97-ad3f-4cbf-86e8-fb0a9146631e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21699e97-ad3f-4cbf-86e8-fb0a9146631e" (UID: "21699e97-ad3f-4cbf-86e8-fb0a9146631e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:39:22.369517 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:22.369494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21699e97-ad3f-4cbf-86e8-fb0a9146631e-kube-api-access-hh9vl" (OuterVolumeSpecName: "kube-api-access-hh9vl") pod "21699e97-ad3f-4cbf-86e8-fb0a9146631e" (UID: "21699e97-ad3f-4cbf-86e8-fb0a9146631e"). InnerVolumeSpecName "kube-api-access-hh9vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:39:22.467893 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:22.467864    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21699e97-ad3f-4cbf-86e8-fb0a9146631e-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:39:22.467893 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:22.467888    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh9vl\" (UniqueName: \"kubernetes.io/projected/21699e97-ad3f-4cbf-86e8-fb0a9146631e-kube-api-access-hh9vl\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:39:23.243039 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:23.242997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9" event={"ID":"21699e97-ad3f-4cbf-86e8-fb0a9146631e","Type":"ContainerDied","Data":"6863bb115f1a1e782aa9636f6ca6c283f05d012e90fb5c27d8aebe05d87bde96"}
Sep 19 08:39:23.243547 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:23.243111    2577 scope.go:117] "RemoveContainer" containerID="a2e5f62b73aae543bfcca7ea79362c84e883e7b45863c0ce9fc51b9d01cb6a03"
Sep 19 08:39:23.243547 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:23.243299    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"
Sep 19 08:39:23.274876 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:23.274849    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"]
Sep 19 08:39:23.282514 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:23.282487    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5cc2l9"]
Sep 19 08:39:24.503938 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:24.503908    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" path="/var/lib/kubelet/pods/21699e97-ad3f-4cbf-86e8-fb0a9146631e/volumes"
Sep 19 08:39:58.878679 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.878598    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"]
Sep 19 08:39:58.879101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.878986    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main"
Sep 19 08:39:58.879101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879000    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main"
Sep 19 08:39:58.879101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879009    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main"
Sep 19 08:39:58.879101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main"
Sep 19 08:39:58.879101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879024    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a895edf3-4891-4a7f-95c1-91af7e48af07" containerName="main"
Sep 19 08:39:58.879101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879030    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a895edf3-4891-4a7f-95c1-91af7e48af07" containerName="main"
Sep 19 08:39:58.879101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879050    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="storage-initializer"
Sep 19 08:39:58.879101 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879057    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="storage-initializer"
Sep 19 08:39:58.879343 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879107    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a895edf3-4891-4a7f-95c1-91af7e48af07" containerName="main"
Sep 19 08:39:58.879343 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879114    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="21699e97-ad3f-4cbf-86e8-fb0a9146631e" containerName="main"
Sep 19 08:39:58.879343 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.879120    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba4c137-22dc-4b19-8920-280f98a0f067" containerName="main"
Sep 19 08:39:58.882065 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.882049    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:58.884779 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.884758    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 19 08:39:58.885858 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.885835    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-s78g4\""
Sep 19 08:39:58.885966 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.885857    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 19 08:39:58.886130 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.886115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 19 08:39:58.891907 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.891886    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"]
Sep 19 08:39:58.965114 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.965082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba67ef7-dbd9-4e08-8289-6984722a75b2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:58.965277 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.965128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:58.965277 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.965210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:58.965277 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.965259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:58.965475 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.965342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkpwg\" (UniqueName: \"kubernetes.io/projected/cba67ef7-dbd9-4e08-8289-6984722a75b2-kube-api-access-zkpwg\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:58.965475 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:58.965400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.066549 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.066513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkpwg\" (UniqueName: \"kubernetes.io/projected/cba67ef7-dbd9-4e08-8289-6984722a75b2-kube-api-access-zkpwg\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.066733 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.066559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.066733 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.066590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba67ef7-dbd9-4e08-8289-6984722a75b2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.066733 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.066610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.066733 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.066638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.066733 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.066666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.067013 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.066992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.067076 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.067035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.067128 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.067069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.068836 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.068807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.069200 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.069178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba67ef7-dbd9-4e08-8289-6984722a75b2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.074600 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.074582    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkpwg\" (UniqueName: \"kubernetes.io/projected/cba67ef7-dbd9-4e08-8289-6984722a75b2-kube-api-access-zkpwg\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-fjx22\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.192182 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.192108    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:39:59.198634 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.198610    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"]
Sep 19 08:39:59.209471 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.209447    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.212069 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.212043    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"]
Sep 19 08:39:59.212181 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.212157    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-hr6b7\""
Sep 19 08:39:59.267951 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.267922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.268084 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.268012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.268158 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.268099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e05b5f-ae87-4ebd-acd5-9a522b998747-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.268158 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.268126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slp77\" (UniqueName: \"kubernetes.io/projected/18e05b5f-ae87-4ebd-acd5-9a522b998747-kube-api-access-slp77\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.369279 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.369245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e05b5f-ae87-4ebd-acd5-9a522b998747-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.369279 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.369280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slp77\" (UniqueName: \"kubernetes.io/projected/18e05b5f-ae87-4ebd-acd5-9a522b998747-kube-api-access-slp77\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.369511 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.369299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.369511 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.369395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.369679 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.369664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.369763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.369743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.371647 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.371632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e05b5f-ae87-4ebd-acd5-9a522b998747-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.377152 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.377135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slp77\" (UniqueName: \"kubernetes.io/projected/18e05b5f-ae87-4ebd-acd5-9a522b998747-kube-api-access-slp77\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.469356 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.469327    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"]
Sep 19 08:39:59.471326 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:39:59.471302    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba67ef7_dbd9_4e08_8289_6984722a75b2.slice/crio-b6efa1e3d1fb415a0fff1f63cd6ff9fe8965627127b3f4033ab15bfc781a0dfe WatchSource:0}: Error finding container b6efa1e3d1fb415a0fff1f63cd6ff9fe8965627127b3f4033ab15bfc781a0dfe: Status 404 returned error can't find the container with id b6efa1e3d1fb415a0fff1f63cd6ff9fe8965627127b3f4033ab15bfc781a0dfe
Sep 19 08:39:59.527812 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.527783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:39:59.800537 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:39:59.800511    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"]
Sep 19 08:39:59.802536 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:39:59.802503    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e05b5f_ae87_4ebd_acd5_9a522b998747.slice/crio-dc3067a94254234e52b28ce0d9fcf7b05d3094fbfd6553e0f9473816487575ef WatchSource:0}: Error finding container dc3067a94254234e52b28ce0d9fcf7b05d3094fbfd6553e0f9473816487575ef: Status 404 returned error can't find the container with id dc3067a94254234e52b28ce0d9fcf7b05d3094fbfd6553e0f9473816487575ef
Sep 19 08:40:00.441783 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:00.441745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22" event={"ID":"cba67ef7-dbd9-4e08-8289-6984722a75b2","Type":"ContainerStarted","Data":"ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7"}
Sep 19 08:40:00.442220 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:00.441803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22" event={"ID":"cba67ef7-dbd9-4e08-8289-6984722a75b2","Type":"ContainerStarted","Data":"b6efa1e3d1fb415a0fff1f63cd6ff9fe8965627127b3f4033ab15bfc781a0dfe"}
Sep 19 08:40:00.443406 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:00.443357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" event={"ID":"18e05b5f-ae87-4ebd-acd5-9a522b998747","Type":"ContainerStarted","Data":"70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf"}
Sep 19 08:40:00.443514 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:00.443425    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:40:00.443514 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:00.443441    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" event={"ID":"18e05b5f-ae87-4ebd-acd5-9a522b998747","Type":"ContainerStarted","Data":"dc3067a94254234e52b28ce0d9fcf7b05d3094fbfd6553e0f9473816487575ef"}
Sep 19 08:40:00.459901 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:00.459858    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22" podStartSLOduration=2.459845545 podStartE2EDuration="2.459845545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:40:00.457445266 +0000 UTC m=+1168.139028698" watchObservedRunningTime="2026-09-19 08:40:00.459845545 +0000 UTC m=+1168.141428976"
Sep 19 08:40:00.477673 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:00.477630    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" podStartSLOduration=1.477618384 podStartE2EDuration="1.477618384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:40:00.475261918 +0000 UTC m=+1168.156845363" watchObservedRunningTime="2026-09-19 08:40:00.477618384 +0000 UTC m=+1168.159201816"
Sep 19 08:40:01.192348 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:01.192315    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:40:09.192517 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:09.192480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:40:09.204720 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:09.204691    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:40:09.506530 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:09.506452    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:40:31.450959 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:31.450925    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:40:33.032834 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.032800    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"]
Sep 19 08:40:33.037461 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.037421    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"]
Sep 19 08:40:33.043172 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:40:33.043126    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:40:33.043172 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:40:33.043172 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:40:33.043172 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7" cmd=["/bin/sleep","15"]
Sep 19 08:40:33.043446 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.043193    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22" message="" err=<
Sep 19 08:40:33.043446 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:40:33.043446 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:40:33.043446 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:40:33.043446 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:40:33.043210    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:40:33.043446 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:40:33.043446 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:40:33.043446 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22" podUID="cba67ef7-dbd9-4e08-8289-6984722a75b2" containerName="main" containerID="cri-o://ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7"
Sep 19 08:40:33.043446 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.043252    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22" podUID="cba67ef7-dbd9-4e08-8289-6984722a75b2" containerName="main" containerID="cri-o://ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7" gracePeriod=60
Sep 19 08:40:33.394929 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.394900    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:40:33.451625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.451596    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkpwg\" (UniqueName: \"kubernetes.io/projected/cba67ef7-dbd9-4e08-8289-6984722a75b2-kube-api-access-zkpwg\") pod \"cba67ef7-dbd9-4e08-8289-6984722a75b2\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") "
Sep 19 08:40:33.451767 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.451675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-tmp-dir\") pod \"cba67ef7-dbd9-4e08-8289-6984722a75b2\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") "
Sep 19 08:40:33.451767 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.451706    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-dshm\") pod \"cba67ef7-dbd9-4e08-8289-6984722a75b2\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") "
Sep 19 08:40:33.451767 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.451733    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-home\") pod \"cba67ef7-dbd9-4e08-8289-6984722a75b2\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") "
Sep 19 08:40:33.451911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.451773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-model-cache\") pod \"cba67ef7-dbd9-4e08-8289-6984722a75b2\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") "
Sep 19 08:40:33.451911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.451816    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba67ef7-dbd9-4e08-8289-6984722a75b2-tls-certs\") pod \"cba67ef7-dbd9-4e08-8289-6984722a75b2\" (UID: \"cba67ef7-dbd9-4e08-8289-6984722a75b2\") "
Sep 19 08:40:33.452016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.451938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cba67ef7-dbd9-4e08-8289-6984722a75b2" (UID: "cba67ef7-dbd9-4e08-8289-6984722a75b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:33.452016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.451965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-home" (OuterVolumeSpecName: "home") pod "cba67ef7-dbd9-4e08-8289-6984722a75b2" (UID: "cba67ef7-dbd9-4e08-8289-6984722a75b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:33.452123 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.452066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cba67ef7-dbd9-4e08-8289-6984722a75b2" (UID: "cba67ef7-dbd9-4e08-8289-6984722a75b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:33.452123 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.452088    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:40:33.452123 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.452113    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:40:33.453818 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.453786    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba67ef7-dbd9-4e08-8289-6984722a75b2-kube-api-access-zkpwg" (OuterVolumeSpecName: "kube-api-access-zkpwg") pod "cba67ef7-dbd9-4e08-8289-6984722a75b2" (UID: "cba67ef7-dbd9-4e08-8289-6984722a75b2"). InnerVolumeSpecName "kube-api-access-zkpwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:40:33.454169 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.454146    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-dshm" (OuterVolumeSpecName: "dshm") pod "cba67ef7-dbd9-4e08-8289-6984722a75b2" (UID: "cba67ef7-dbd9-4e08-8289-6984722a75b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:33.454169 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.454149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba67ef7-dbd9-4e08-8289-6984722a75b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cba67ef7-dbd9-4e08-8289-6984722a75b2" (UID: "cba67ef7-dbd9-4e08-8289-6984722a75b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:40:33.552974 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.552950    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba67ef7-dbd9-4e08-8289-6984722a75b2-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:40:33.552974 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.552971    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkpwg\" (UniqueName: \"kubernetes.io/projected/cba67ef7-dbd9-4e08-8289-6984722a75b2-kube-api-access-zkpwg\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:40:33.552974 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.552980    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:40:33.553161 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.552989    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba67ef7-dbd9-4e08-8289-6984722a75b2-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:40:33.623736 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.623708    2577 generic.go:356] "Generic (PLEG): container finished" podID="cba67ef7-dbd9-4e08-8289-6984722a75b2" containerID="ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7" exitCode=0
Sep 19 08:40:33.623867 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.623760    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"
Sep 19 08:40:33.623867 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.623801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22" event={"ID":"cba67ef7-dbd9-4e08-8289-6984722a75b2","Type":"ContainerDied","Data":"ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7"}
Sep 19 08:40:33.623992 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.623866    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22" event={"ID":"cba67ef7-dbd9-4e08-8289-6984722a75b2","Type":"ContainerDied","Data":"b6efa1e3d1fb415a0fff1f63cd6ff9fe8965627127b3f4033ab15bfc781a0dfe"}
Sep 19 08:40:33.623992 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.623895    2577 scope.go:117] "RemoveContainer" containerID="ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7"
Sep 19 08:40:33.633104 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.633087    2577 scope.go:117] "RemoveContainer" containerID="ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7"
Sep 19 08:40:33.633353 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:40:33.633335    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7\": container with ID starting with ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7 not found: ID does not exist" containerID="ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7"
Sep 19 08:40:33.633411 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.633376    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7"} err="failed to get container status \"ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7\": rpc error: code = NotFound desc = could not find container \"ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7\": container with ID starting with ef2327d2e35b1f74969bad3fcd15bf9604444d363b684f0f938a5f44daee81c7 not found: ID does not exist"
Sep 19 08:40:33.646238 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.645678    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"]
Sep 19 08:40:33.648440 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:33.648419    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-fjx22"]
Sep 19 08:40:34.504975 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:34.504944    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cba67ef7-dbd9-4e08-8289-6984722a75b2" path="/var/lib/kubelet/pods/cba67ef7-dbd9-4e08-8289-6984722a75b2/volumes"
Sep 19 08:40:34.518932 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:34.518908    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:40:34.524561 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:34.524546    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:40:41.450696 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:41.450654    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" podUID="18e05b5f-ae87-4ebd-acd5-9a522b998747" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:40:48.057075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:48.057029    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" podUID="18e05b5f-ae87-4ebd-acd5-9a522b998747" containerName="main" containerID="cri-o://70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf" gracePeriod=45
Sep 19 08:40:51.450613 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:40:51.450571    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" podUID="18e05b5f-ae87-4ebd-acd5-9a522b998747" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:41:01.450272 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:01.450235    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" podUID="18e05b5f-ae87-4ebd-acd5-9a522b998747" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:41:01.450762 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:01.450392    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:41:11.451174 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:11.451131    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" podUID="18e05b5f-ae87-4ebd-acd5-9a522b998747" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:41:18.306569 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.306515    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:41:18.333245 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.333159    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slp77\" (UniqueName: \"kubernetes.io/projected/18e05b5f-ae87-4ebd-acd5-9a522b998747-kube-api-access-slp77\") pod \"18e05b5f-ae87-4ebd-acd5-9a522b998747\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") "
Sep 19 08:41:18.333245 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.333215    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-tokenizer-cache\") pod \"18e05b5f-ae87-4ebd-acd5-9a522b998747\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") "
Sep 19 08:41:18.333551 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.333264    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e05b5f-ae87-4ebd-acd5-9a522b998747-tls-certs\") pod \"18e05b5f-ae87-4ebd-acd5-9a522b998747\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") "
Sep 19 08:41:18.333551 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.333311    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-cachi2-cache\") pod \"18e05b5f-ae87-4ebd-acd5-9a522b998747\" (UID: \"18e05b5f-ae87-4ebd-acd5-9a522b998747\") "
Sep 19 08:41:18.333551 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.333481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "18e05b5f-ae87-4ebd-acd5-9a522b998747" (UID: "18e05b5f-ae87-4ebd-acd5-9a522b998747"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:41:18.333659 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.333562    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-tokenizer-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:41:18.333659 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.333567    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "18e05b5f-ae87-4ebd-acd5-9a522b998747" (UID: "18e05b5f-ae87-4ebd-acd5-9a522b998747"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:41:18.335446 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.335416    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e05b5f-ae87-4ebd-acd5-9a522b998747-kube-api-access-slp77" (OuterVolumeSpecName: "kube-api-access-slp77") pod "18e05b5f-ae87-4ebd-acd5-9a522b998747" (UID: "18e05b5f-ae87-4ebd-acd5-9a522b998747"). InnerVolumeSpecName "kube-api-access-slp77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:41:18.335446 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.335435    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e05b5f-ae87-4ebd-acd5-9a522b998747-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18e05b5f-ae87-4ebd-acd5-9a522b998747" (UID: "18e05b5f-ae87-4ebd-acd5-9a522b998747"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:41:18.434794 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.434755    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e05b5f-ae87-4ebd-acd5-9a522b998747-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:41:18.434794 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.434783    2577 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e05b5f-ae87-4ebd-acd5-9a522b998747-cachi2-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:41:18.434794 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.434793    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slp77\" (UniqueName: \"kubernetes.io/projected/18e05b5f-ae87-4ebd-acd5-9a522b998747-kube-api-access-slp77\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:41:18.853161 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.853127    2577 generic.go:356] "Generic (PLEG): container finished" podID="18e05b5f-ae87-4ebd-acd5-9a522b998747" containerID="70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf" exitCode=0
Sep 19 08:41:18.853326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.853187    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" event={"ID":"18e05b5f-ae87-4ebd-acd5-9a522b998747","Type":"ContainerDied","Data":"70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf"}
Sep 19 08:41:18.853326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.853193    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"
Sep 19 08:41:18.853326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.853232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq" event={"ID":"18e05b5f-ae87-4ebd-acd5-9a522b998747","Type":"ContainerDied","Data":"dc3067a94254234e52b28ce0d9fcf7b05d3094fbfd6553e0f9473816487575ef"}
Sep 19 08:41:18.853326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.853253    2577 scope.go:117] "RemoveContainer" containerID="70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf"
Sep 19 08:41:18.862184 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.862165    2577 scope.go:117] "RemoveContainer" containerID="70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf"
Sep 19 08:41:18.862478 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:41:18.862457    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf\": container with ID starting with 70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf not found: ID does not exist" containerID="70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf"
Sep 19 08:41:18.862580 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.862485    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf"} err="failed to get container status \"70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf\": rpc error: code = NotFound desc = could not find container \"70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf\": container with ID starting with 70eee2c99af2632fdf1509ac8e32a37699ffd83a3a59c115a51da1d833de5ecf not found: ID does not exist"
Sep 19 08:41:18.872901 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.872876    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"]
Sep 19 08:41:18.876813 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:18.876789    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcnx7cq"]
Sep 19 08:41:20.172691 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.172654    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"]
Sep 19 08:41:20.173074 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.173044    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba67ef7-dbd9-4e08-8289-6984722a75b2" containerName="main"
Sep 19 08:41:20.173074 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.173056    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba67ef7-dbd9-4e08-8289-6984722a75b2" containerName="main"
Sep 19 08:41:20.173164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.173076    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e05b5f-ae87-4ebd-acd5-9a522b998747" containerName="main"
Sep 19 08:41:20.173164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.173082    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e05b5f-ae87-4ebd-acd5-9a522b998747" containerName="main"
Sep 19 08:41:20.173164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.173155    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba67ef7-dbd9-4e08-8289-6984722a75b2" containerName="main"
Sep 19 08:41:20.173164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.173164    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e05b5f-ae87-4ebd-acd5-9a522b998747" containerName="main"
Sep 19 08:41:20.177682 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.177660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.181543 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.181518    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 19 08:41:20.181704 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.181552    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 19 08:41:20.181704 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.181525    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 19 08:41:20.181704 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.181605    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-xjmgp\""
Sep 19 08:41:20.187626 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.187598    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"]
Sep 19 08:41:20.249228 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.249191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-home\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.249228 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.249238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.249501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.249315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce545d-d689-41e5-bc3c-96221963b788-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.249501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.249403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wng5r\" (UniqueName: \"kubernetes.io/projected/93ce545d-d689-41e5-bc3c-96221963b788-kube-api-access-wng5r\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.249501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.249433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.249501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.249475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.350229 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.350191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.350437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.350283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-home\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.350437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.350313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.350437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.350344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce545d-d689-41e5-bc3c-96221963b788-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.350437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.350397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wng5r\" (UniqueName: \"kubernetes.io/projected/93ce545d-d689-41e5-bc3c-96221963b788-kube-api-access-wng5r\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.350437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.350425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.350749 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.350714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.350813 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.350748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-home\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.350813 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.350762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.352703 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.352677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.352977 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.352947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce545d-d689-41e5-bc3c-96221963b788-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.358479 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.358453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wng5r\" (UniqueName: \"kubernetes.io/projected/93ce545d-d689-41e5-bc3c-96221963b788-kube-api-access-wng5r\") pod \"fc-concurrency-test-kserve-b5d44467c-jqhhh\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.490525 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.490424    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:20.505122 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.505089    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e05b5f-ae87-4ebd-acd5-9a522b998747" path="/var/lib/kubelet/pods/18e05b5f-ae87-4ebd-acd5-9a522b998747/volumes"
Sep 19 08:41:20.766954 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.766920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"]
Sep 19 08:41:20.769317 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:41:20.769293    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ce545d_d689_41e5_bc3c_96221963b788.slice/crio-1e05e3ec4d452dc2a595ec9e3af77d695091db1390dc3d196f343a09099a2ea9 WatchSource:0}: Error finding container 1e05e3ec4d452dc2a595ec9e3af77d695091db1390dc3d196f343a09099a2ea9: Status 404 returned error can't find the container with id 1e05e3ec4d452dc2a595ec9e3af77d695091db1390dc3d196f343a09099a2ea9
Sep 19 08:41:20.867665 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.867634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh" event={"ID":"93ce545d-d689-41e5-bc3c-96221963b788","Type":"ContainerStarted","Data":"e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9"}
Sep 19 08:41:20.867807 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.867682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh" event={"ID":"93ce545d-d689-41e5-bc3c-96221963b788","Type":"ContainerStarted","Data":"1e05e3ec4d452dc2a595ec9e3af77d695091db1390dc3d196f343a09099a2ea9"}
Sep 19 08:41:20.891869 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:20.891813    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh" podStartSLOduration=0.891796604 podStartE2EDuration="891.796604ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:41:20.890951804 +0000 UTC m=+1248.572535238" watchObservedRunningTime="2026-09-19 08:41:20.891796604 +0000 UTC m=+1248.573380037"
Sep 19 08:41:21.490909 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:21.490873    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:30.490901 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:30.490868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:30.508855 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:30.508826    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:30.932950 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:30.932917    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:41:44.134550 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.134511    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"]
Sep 19 08:41:44.138994 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.138965    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.142374 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.142335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 19 08:41:44.143354 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.143326    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-c787g\""
Sep 19 08:41:44.143514 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.143387    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 19 08:41:44.143514 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.143399    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 19 08:41:44.160464 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.160425    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"]
Sep 19 08:41:44.258094 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.258062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.258287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.258099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.258287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.258128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.258287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.258156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584jt\" (UniqueName: \"kubernetes.io/projected/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-kube-api-access-584jt\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.258287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.258193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.258287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.258250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.359389 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.359321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.359389 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.359384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.359614 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.359405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.359614 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.359536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.359614 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.359586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584jt\" (UniqueName: \"kubernetes.io/projected/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-kube-api-access-584jt\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.359734 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.359622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.359805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.359788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.359867 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.359838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.359943 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.359918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.361711 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.361685    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.361891 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.361874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.369001 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.368974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-584jt\" (UniqueName: \"kubernetes.io/projected/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-kube-api-access-584jt\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.449998 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.449905    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:44.723504 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:44.723471    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"]
Sep 19 08:41:44.725011 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:41:44.724974    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b66d779_fb8c_4a21_b4a6_752a27b15e6c.slice/crio-549cbaa290acf52e11c0112bc2444188e66623cd86ef310e9127a33bc082a99e WatchSource:0}: Error finding container 549cbaa290acf52e11c0112bc2444188e66623cd86ef310e9127a33bc082a99e: Status 404 returned error can't find the container with id 549cbaa290acf52e11c0112bc2444188e66623cd86ef310e9127a33bc082a99e
Sep 19 08:41:45.001553 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:45.001457    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv" event={"ID":"6b66d779-fb8c-4a21-b4a6-752a27b15e6c","Type":"ContainerStarted","Data":"66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9"}
Sep 19 08:41:45.001553 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:45.001510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv" event={"ID":"6b66d779-fb8c-4a21-b4a6-752a27b15e6c","Type":"ContainerStarted","Data":"549cbaa290acf52e11c0112bc2444188e66623cd86ef310e9127a33bc082a99e"}
Sep 19 08:41:45.019937 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:45.019882    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv" podStartSLOduration=1.019866862 podStartE2EDuration="1.019866862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:41:45.018519546 +0000 UTC m=+1272.700102980" watchObservedRunningTime="2026-09-19 08:41:45.019866862 +0000 UTC m=+1272.701450294"
Sep 19 08:41:45.450827 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:45.450787    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:54.450862 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:54.450828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:54.463906 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:54.463874    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:41:55.067889 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:41:55.067852    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:42:01.004183 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.004147    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"]
Sep 19 08:42:01.016912 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:42:01.016870    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:42:01.016912 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:42:01.016912 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:42:01.016912 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9" cmd=["/bin/sleep","15"]
Sep 19 08:42:01.017270 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.016937    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh" message="" err=<
Sep 19 08:42:01.017270 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:42:01.017270 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:42:01.017270 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:42:01.017270 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:42:01.016954    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:42:01.017270 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:42:01.017270 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:42:01.017270 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh" podUID="93ce545d-d689-41e5-bc3c-96221963b788" containerName="main" containerID="cri-o://e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9"
Sep 19 08:42:01.017270 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.017017    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh" podUID="93ce545d-d689-41e5-bc3c-96221963b788" containerName="main" containerID="cri-o://e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9" gracePeriod=60
Sep 19 08:42:01.762915 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.762882    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:42:01.922012 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.921922    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce545d-d689-41e5-bc3c-96221963b788-tls-certs\") pod \"93ce545d-d689-41e5-bc3c-96221963b788\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") "
Sep 19 08:42:01.922012 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.921979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-home\") pod \"93ce545d-d689-41e5-bc3c-96221963b788\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") "
Sep 19 08:42:01.922249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.922024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wng5r\" (UniqueName: \"kubernetes.io/projected/93ce545d-d689-41e5-bc3c-96221963b788-kube-api-access-wng5r\") pod \"93ce545d-d689-41e5-bc3c-96221963b788\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") "
Sep 19 08:42:01.922249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.922064    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-dshm\") pod \"93ce545d-d689-41e5-bc3c-96221963b788\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") "
Sep 19 08:42:01.922249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.922081    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-tmp-dir\") pod \"93ce545d-d689-41e5-bc3c-96221963b788\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") "
Sep 19 08:42:01.922249 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.922113    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-model-cache\") pod \"93ce545d-d689-41e5-bc3c-96221963b788\" (UID: \"93ce545d-d689-41e5-bc3c-96221963b788\") "
Sep 19 08:42:01.922509 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.922307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-home" (OuterVolumeSpecName: "home") pod "93ce545d-d689-41e5-bc3c-96221963b788" (UID: "93ce545d-d689-41e5-bc3c-96221963b788"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:01.922509 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.922334    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93ce545d-d689-41e5-bc3c-96221963b788" (UID: "93ce545d-d689-41e5-bc3c-96221963b788"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:01.922583 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.922535    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-model-cache" (OuterVolumeSpecName: "model-cache") pod "93ce545d-d689-41e5-bc3c-96221963b788" (UID: "93ce545d-d689-41e5-bc3c-96221963b788"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:01.924408 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.924380    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-dshm" (OuterVolumeSpecName: "dshm") pod "93ce545d-d689-41e5-bc3c-96221963b788" (UID: "93ce545d-d689-41e5-bc3c-96221963b788"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:01.924529 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.924452    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ce545d-d689-41e5-bc3c-96221963b788-kube-api-access-wng5r" (OuterVolumeSpecName: "kube-api-access-wng5r") pod "93ce545d-d689-41e5-bc3c-96221963b788" (UID: "93ce545d-d689-41e5-bc3c-96221963b788"). InnerVolumeSpecName "kube-api-access-wng5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:42:01.924581 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:01.924550    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ce545d-d689-41e5-bc3c-96221963b788-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93ce545d-d689-41e5-bc3c-96221963b788" (UID: "93ce545d-d689-41e5-bc3c-96221963b788"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:42:02.023532 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.023498    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93ce545d-d689-41e5-bc3c-96221963b788-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:02.023532 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.023525    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:02.023532 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.023534    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wng5r\" (UniqueName: \"kubernetes.io/projected/93ce545d-d689-41e5-bc3c-96221963b788-kube-api-access-wng5r\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:02.023532 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.023544    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:02.024007 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.023552    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:02.024007 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.023559    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93ce545d-d689-41e5-bc3c-96221963b788-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:02.095877 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.095844    2577 generic.go:356] "Generic (PLEG): container finished" podID="93ce545d-d689-41e5-bc3c-96221963b788" containerID="e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9" exitCode=0
Sep 19 08:42:02.096023 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.095894    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"
Sep 19 08:42:02.096023 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.095927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh" event={"ID":"93ce545d-d689-41e5-bc3c-96221963b788","Type":"ContainerDied","Data":"e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9"}
Sep 19 08:42:02.096023 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.096012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh" event={"ID":"93ce545d-d689-41e5-bc3c-96221963b788","Type":"ContainerDied","Data":"1e05e3ec4d452dc2a595ec9e3af77d695091db1390dc3d196f343a09099a2ea9"}
Sep 19 08:42:02.096143 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.096039    2577 scope.go:117] "RemoveContainer" containerID="e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9"
Sep 19 08:42:02.105757 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.105732    2577 scope.go:117] "RemoveContainer" containerID="e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9"
Sep 19 08:42:02.106073 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:42:02.106050    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9\": container with ID starting with e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9 not found: ID does not exist" containerID="e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9"
Sep 19 08:42:02.106149 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.106083    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9"} err="failed to get container status \"e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9\": rpc error: code = NotFound desc = could not find container \"e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9\": container with ID starting with e8398f110947b4792e952fa1247dca372726492a8f23b3b7fee8d74aaee48ee9 not found: ID does not exist"
Sep 19 08:42:02.117779 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.117747    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"]
Sep 19 08:42:02.124281 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.124254    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-jqhhh"]
Sep 19 08:42:02.504277 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:02.504233    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93ce545d-d689-41e5-bc3c-96221963b788" path="/var/lib/kubelet/pods/93ce545d-d689-41e5-bc3c-96221963b788/volumes"
Sep 19 08:42:22.435000 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.434960    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"]
Sep 19 08:42:22.444313 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:42:22.444272    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:42:22.444313 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:42:22.444313 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:42:22.444313 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9" cmd=["/bin/sleep","15"]
Sep 19 08:42:22.444652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.444335    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv" message="" err=<
Sep 19 08:42:22.444652 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:42:22.444652 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:42:22.444652 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:42:22.444652 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:42:22.444352    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:42:22.444652 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:42:22.444652 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:42:22.444652 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv" podUID="6b66d779-fb8c-4a21-b4a6-752a27b15e6c" containerName="main" containerID="cri-o://66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9"
Sep 19 08:42:22.444652 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.444419    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv" podUID="6b66d779-fb8c-4a21-b4a6-752a27b15e6c" containerName="main" containerID="cri-o://66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9" gracePeriod=60
Sep 19 08:42:22.799632 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.799605    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:42:22.896559 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.896529    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tls-certs\") pod \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") "
Sep 19 08:42:22.896763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.896585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-dshm\") pod \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") "
Sep 19 08:42:22.896763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.896633    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tmp-dir\") pod \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") "
Sep 19 08:42:22.896763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.896656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-584jt\" (UniqueName: \"kubernetes.io/projected/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-kube-api-access-584jt\") pod \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") "
Sep 19 08:42:22.896763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.896685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-model-cache\") pod \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") "
Sep 19 08:42:22.896763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.896720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-home\") pod \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\" (UID: \"6b66d779-fb8c-4a21-b4a6-752a27b15e6c\") "
Sep 19 08:42:22.897045 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.896911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b66d779-fb8c-4a21-b4a6-752a27b15e6c" (UID: "6b66d779-fb8c-4a21-b4a6-752a27b15e6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:22.897045 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.896950    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b66d779-fb8c-4a21-b4a6-752a27b15e6c" (UID: "6b66d779-fb8c-4a21-b4a6-752a27b15e6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:22.897045 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.897034    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-home" (OuterVolumeSpecName: "home") pod "6b66d779-fb8c-4a21-b4a6-752a27b15e6c" (UID: "6b66d779-fb8c-4a21-b4a6-752a27b15e6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:22.897202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.897042    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:22.897202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.897070    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:22.898936 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.898911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-dshm" (OuterVolumeSpecName: "dshm") pod "6b66d779-fb8c-4a21-b4a6-752a27b15e6c" (UID: "6b66d779-fb8c-4a21-b4a6-752a27b15e6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:22.899048 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.898988    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b66d779-fb8c-4a21-b4a6-752a27b15e6c" (UID: "6b66d779-fb8c-4a21-b4a6-752a27b15e6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:42:22.899048 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.899003    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-kube-api-access-584jt" (OuterVolumeSpecName: "kube-api-access-584jt") pod "6b66d779-fb8c-4a21-b4a6-752a27b15e6c" (UID: "6b66d779-fb8c-4a21-b4a6-752a27b15e6c"). InnerVolumeSpecName "kube-api-access-584jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:42:22.998020 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.997928    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:22.998020 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.997959    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-584jt\" (UniqueName: \"kubernetes.io/projected/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-kube-api-access-584jt\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:22.998020 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.997972    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:22.998020 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:22.997980    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b66d779-fb8c-4a21-b4a6-752a27b15e6c-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:42:23.210323 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:23.210287    2577 generic.go:356] "Generic (PLEG): container finished" podID="6b66d779-fb8c-4a21-b4a6-752a27b15e6c" containerID="66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9" exitCode=0
Sep 19 08:42:23.210323 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:23.210329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv" event={"ID":"6b66d779-fb8c-4a21-b4a6-752a27b15e6c","Type":"ContainerDied","Data":"66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9"}
Sep 19 08:42:23.210572 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:23.210338    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"
Sep 19 08:42:23.210572 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:23.210391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv" event={"ID":"6b66d779-fb8c-4a21-b4a6-752a27b15e6c","Type":"ContainerDied","Data":"549cbaa290acf52e11c0112bc2444188e66623cd86ef310e9127a33bc082a99e"}
Sep 19 08:42:23.210572 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:23.210415    2577 scope.go:117] "RemoveContainer" containerID="66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9"
Sep 19 08:42:23.222832 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:23.222807    2577 scope.go:117] "RemoveContainer" containerID="66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9"
Sep 19 08:42:23.223131 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:42:23.223111    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9\": container with ID starting with 66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9 not found: ID does not exist" containerID="66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9"
Sep 19 08:42:23.223208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:23.223138    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9"} err="failed to get container status \"66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9\": rpc error: code = NotFound desc = could not find container \"66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9\": container with ID starting with 66240b2a1e0db42c9bccc0bb81c95e7d132eeb834fd11e94a359ed3595bd47c9 not found: ID does not exist"
Sep 19 08:42:23.235632 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:23.235602    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"]
Sep 19 08:42:23.238791 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:23.238768    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-szpfv"]
Sep 19 08:42:24.504288 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:24.504255    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b66d779-fb8c-4a21-b4a6-752a27b15e6c" path="/var/lib/kubelet/pods/6b66d779-fb8c-4a21-b4a6-752a27b15e6c/volumes"
Sep 19 08:42:51.403577 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.403477    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"]
Sep 19 08:42:51.404040 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.404018    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93ce545d-d689-41e5-bc3c-96221963b788" containerName="main"
Sep 19 08:42:51.404040 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.404033    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="93ce545d-d689-41e5-bc3c-96221963b788" containerName="main"
Sep 19 08:42:51.404167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.404046    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b66d779-fb8c-4a21-b4a6-752a27b15e6c" containerName="main"
Sep 19 08:42:51.404167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.404054    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b66d779-fb8c-4a21-b4a6-752a27b15e6c" containerName="main"
Sep 19 08:42:51.404167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.404152    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="93ce545d-d689-41e5-bc3c-96221963b788" containerName="main"
Sep 19 08:42:51.404167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.404164    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b66d779-fb8c-4a21-b4a6-752a27b15e6c" containerName="main"
Sep 19 08:42:51.408991 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.408964    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.411672 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.411649    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 19 08:42:51.411802 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.411739    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-x4htx\""
Sep 19 08:42:51.411997 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.411974    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 19 08:42:51.412119 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.412033    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 19 08:42:51.419221 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.419198    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"]
Sep 19 08:42:51.559824 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.559785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-credential-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.559998 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.559831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-data\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.559998 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.559856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-istio-token\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.559998 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.559908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-certs\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.559998 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.559924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c398b03f-2733-46f0-a128-e94da065630a-istio-podinfo\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.560164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.559989    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c398b03f-2733-46f0-a128-e94da065630a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.560164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.560020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.560164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.560037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-envoy\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.560164 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.560063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvtb4\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-kube-api-access-rvtb4\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.660999 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.660895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-data\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.660999 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.660936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-istio-token\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-certs\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c398b03f-2733-46f0-a128-e94da065630a-istio-podinfo\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c398b03f-2733-46f0-a128-e94da065630a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661531 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-data\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661531 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-certs\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661531 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-envoy\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661699 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661699 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvtb4\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-kube-api-access-rvtb4\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.661699 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.661626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-credential-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.662035 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.662009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-credential-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.662202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.662178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c398b03f-2733-46f0-a128-e94da065630a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.663923 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.663903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-envoy\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.664045 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.663966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c398b03f-2733-46f0-a128-e94da065630a-istio-podinfo\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.670358 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.670338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-istio-token\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.670765 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.670744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvtb4\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-kube-api-access-rvtb4\") pod \"router-gateway-1-openshift-default-6fd8d584bd-xtprz\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:51.724214 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:51.724185    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:52.004739 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:52.004711    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"]
Sep 19 08:42:52.006221 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:42:52.006192    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc398b03f_2733_46f0_a128_e94da065630a.slice/crio-8de894c5297307816e3f5aefe5aee76890d0a86ded44dbe42743fc264311b1b2 WatchSource:0}: Error finding container 8de894c5297307816e3f5aefe5aee76890d0a86ded44dbe42743fc264311b1b2: Status 404 returned error can't find the container with id 8de894c5297307816e3f5aefe5aee76890d0a86ded44dbe42743fc264311b1b2
Sep 19 08:42:52.008489 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:52.008453    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:42:52.008614 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:52.008522    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:42:52.008614 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:52.008557    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:42:52.362280 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:52.362242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz" event={"ID":"c398b03f-2733-46f0-a128-e94da065630a","Type":"ContainerStarted","Data":"e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d"}
Sep 19 08:42:52.362532 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:52.362314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz" event={"ID":"c398b03f-2733-46f0-a128-e94da065630a","Type":"ContainerStarted","Data":"8de894c5297307816e3f5aefe5aee76890d0a86ded44dbe42743fc264311b1b2"}
Sep 19 08:42:52.384940 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:52.384896    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz" podStartSLOduration=1.384881877 podStartE2EDuration="1.384881877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:42:52.382889069 +0000 UTC m=+1340.064472503" watchObservedRunningTime="2026-09-19 08:42:52.384881877 +0000 UTC m=+1340.066465340"
Sep 19 08:42:52.725077 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:52.724990    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:53.729638 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:53.729602    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:54.372765 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:54.372717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:42:54.373693 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:42:54.373675    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:43:04.550958 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.550919    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"]
Sep 19 08:43:04.554785 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.554759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.558075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.558050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 19 08:43:04.558075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.558057    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-snnxr\""
Sep 19 08:43:04.564902 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.564879    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"]
Sep 19 08:43:04.675871 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.675837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ff18c0-0673-4bd9-8f95-8188571be9b8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.676042 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.675879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.676042 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.675904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.676042 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.675925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.676042 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.675976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kc8d\" (UniqueName: \"kubernetes.io/projected/09ff18c0-0673-4bd9-8f95-8188571be9b8-kube-api-access-5kc8d\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.676042 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.676002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.776696 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.776661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kc8d\" (UniqueName: \"kubernetes.io/projected/09ff18c0-0673-4bd9-8f95-8188571be9b8-kube-api-access-5kc8d\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.776893 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.776723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.776893 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.776751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ff18c0-0673-4bd9-8f95-8188571be9b8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.776893 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.776791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.776893 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.776828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.776893 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.776859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.777225 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.777200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.777298 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.777245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.777298 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.777281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.779127 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.779100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.779330 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.779312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ff18c0-0673-4bd9-8f95-8188571be9b8-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.784286 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.784265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kc8d\" (UniqueName: \"kubernetes.io/projected/09ff18c0-0673-4bd9-8f95-8188571be9b8-kube-api-access-5kc8d\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:04.867234 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:04.867205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:05.142508 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:05.142458    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"]
Sep 19 08:43:05.144695 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:43:05.144665    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ff18c0_0673_4bd9_8f95_8188571be9b8.slice/crio-b11157c4626bb4d96e32edc11c6cdeb8113b969982c0fd871a7fdd8c0ddacd02 WatchSource:0}: Error finding container b11157c4626bb4d96e32edc11c6cdeb8113b969982c0fd871a7fdd8c0ddacd02: Status 404 returned error can't find the container with id b11157c4626bb4d96e32edc11c6cdeb8113b969982c0fd871a7fdd8c0ddacd02
Sep 19 08:43:05.438306 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:05.437948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb" event={"ID":"09ff18c0-0673-4bd9-8f95-8188571be9b8","Type":"ContainerStarted","Data":"064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1"}
Sep 19 08:43:05.438306 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:05.438044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb" event={"ID":"09ff18c0-0673-4bd9-8f95-8188571be9b8","Type":"ContainerStarted","Data":"b11157c4626bb4d96e32edc11c6cdeb8113b969982c0fd871a7fdd8c0ddacd02"}
Sep 19 08:43:05.457621 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:05.457572    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb" podStartSLOduration=1.45755409 podStartE2EDuration="1.45755409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:05.456010808 +0000 UTC m=+1353.137594242" watchObservedRunningTime="2026-09-19 08:43:05.45755409 +0000 UTC m=+1353.139137522"
Sep 19 08:43:05.868013 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:05.867985    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:07.041864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.041829    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"]
Sep 19 08:43:07.045574 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.045533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.048374 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.048338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-rrhxk\""
Sep 19 08:43:07.048561 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.048536    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 19 08:43:07.048703 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.048673    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 19 08:43:07.048827 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.048682    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 19 08:43:07.059096 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.059071    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"]
Sep 19 08:43:07.097023 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.096992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-token\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.097023 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.097025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-credential-socket\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.097285 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.097053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-envoy\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.097285 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.097080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-data\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.097285 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.097106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-podinfo\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.097285 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.097136    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-certs\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.097285 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.097202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.097582 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.097305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-socket\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.097582 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.097343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkk4n\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-kube-api-access-mkk4n\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.198312 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.198555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-socket\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.198555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkk4n\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-kube-api-access-mkk4n\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.198555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-token\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.198555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-credential-socket\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.198555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-envoy\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.198555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-data\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.198901 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-podinfo\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.198901 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-certs\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.199004 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.198924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-credential-socket\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.199070 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.199024    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-data\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.199070 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.199051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-certs\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.199194 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.199178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-socket\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.199256 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.199210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.201068 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.201045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-envoy\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.201195 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.201149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-podinfo\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.206305 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.206278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-token\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.209388 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.206515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkk4n\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-kube-api-access-mkk4n\") pod \"router-gateway-1-openshift-default-758f598944-xhjsf\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.358124 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.358074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:07.661200 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.661174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"]
Sep 19 08:43:07.662219 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:43:07.662187    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c17a777_2b5d_4d0f_9b82_a645a1e2d42f.slice/crio-d95cdc9d30836e2ed6d0f48a8b438d50c7ce36022525c1ef57cb287e03a5de3a WatchSource:0}: Error finding container d95cdc9d30836e2ed6d0f48a8b438d50c7ce36022525c1ef57cb287e03a5de3a: Status 404 returned error can't find the container with id d95cdc9d30836e2ed6d0f48a8b438d50c7ce36022525c1ef57cb287e03a5de3a
Sep 19 08:43:07.664526 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.664482    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:43:07.664592 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.664576    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:43:07.664642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:07.664621    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 19 08:43:08.457631 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:08.457593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf" event={"ID":"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f","Type":"ContainerStarted","Data":"61550cf06492e3c1f8b7678c933c48cea21a8e11344f0bbdab37d6dd5187f5e8"}
Sep 19 08:43:08.458036 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:08.457661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf" event={"ID":"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f","Type":"ContainerStarted","Data":"d95cdc9d30836e2ed6d0f48a8b438d50c7ce36022525c1ef57cb287e03a5de3a"}
Sep 19 08:43:08.491250 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:08.491196    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf" podStartSLOduration=1.491182011 podStartE2EDuration="1.491182011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:08.487898404 +0000 UTC m=+1356.169481837" watchObservedRunningTime="2026-09-19 08:43:08.491182011 +0000 UTC m=+1356.172765509"
Sep 19 08:43:09.358996 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:09.358958    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:09.363927 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:09.363902    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:09.470630 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:09.470600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:09.471709 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:09.471682    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:11.692200 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:11.692153    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"]
Sep 19 08:43:11.697105 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:11.697077    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"]
Sep 19 08:43:11.697310 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:11.697263    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz" podUID="c398b03f-2733-46f0-a128-e94da065630a" containerName="istio-proxy" containerID="cri-o://e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d" gracePeriod=30
Sep 19 08:43:11.702503 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:43:11.702465    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:43:11.702503 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:43:11.702503 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:43:11.702503 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1" cmd=["/bin/sleep","15"]
Sep 19 08:43:11.702705 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:11.702540    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb" message="" err=<
Sep 19 08:43:11.702705 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:43:11.702705 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:43:11.702705 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:43:11.702705 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:43:11.702552    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:43:11.702705 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:43:11.702705 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:43:11.702705 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb" podUID="09ff18c0-0673-4bd9-8f95-8188571be9b8" containerName="main" containerID="cri-o://064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1"
Sep 19 08:43:11.702705 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:11.702591    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb" podUID="09ff18c0-0673-4bd9-8f95-8188571be9b8" containerName="main" containerID="cri-o://064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1" gracePeriod=60
Sep 19 08:43:11.950998 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:11.950973    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:12.052726 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.052689    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-model-cache\") pod \"09ff18c0-0673-4bd9-8f95-8188571be9b8\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") "
Sep 19 08:43:12.052904 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.052751    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-home\") pod \"09ff18c0-0673-4bd9-8f95-8188571be9b8\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") "
Sep 19 08:43:12.052904 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.052796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-dshm\") pod \"09ff18c0-0673-4bd9-8f95-8188571be9b8\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") "
Sep 19 08:43:12.052904 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.052862    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ff18c0-0673-4bd9-8f95-8188571be9b8-tls-certs\") pod \"09ff18c0-0673-4bd9-8f95-8188571be9b8\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") "
Sep 19 08:43:12.052904 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.052883    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-tmp-dir\") pod \"09ff18c0-0673-4bd9-8f95-8188571be9b8\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") "
Sep 19 08:43:12.053131 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.052907    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kc8d\" (UniqueName: \"kubernetes.io/projected/09ff18c0-0673-4bd9-8f95-8188571be9b8-kube-api-access-5kc8d\") pod \"09ff18c0-0673-4bd9-8f95-8188571be9b8\" (UID: \"09ff18c0-0673-4bd9-8f95-8188571be9b8\") "
Sep 19 08:43:12.053131 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.052938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "09ff18c0-0673-4bd9-8f95-8188571be9b8" (UID: "09ff18c0-0673-4bd9-8f95-8188571be9b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:12.053131 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.053006    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-home" (OuterVolumeSpecName: "home") pod "09ff18c0-0673-4bd9-8f95-8188571be9b8" (UID: "09ff18c0-0673-4bd9-8f95-8188571be9b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:12.053288 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.053217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09ff18c0-0673-4bd9-8f95-8188571be9b8" (UID: "09ff18c0-0673-4bd9-8f95-8188571be9b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:12.053288 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.053229    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:12.053288 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.053248    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:12.055141 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.055111    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ff18c0-0673-4bd9-8f95-8188571be9b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09ff18c0-0673-4bd9-8f95-8188571be9b8" (UID: "09ff18c0-0673-4bd9-8f95-8188571be9b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:43:12.055244 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.055175    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ff18c0-0673-4bd9-8f95-8188571be9b8-kube-api-access-5kc8d" (OuterVolumeSpecName: "kube-api-access-5kc8d") pod "09ff18c0-0673-4bd9-8f95-8188571be9b8" (UID: "09ff18c0-0673-4bd9-8f95-8188571be9b8"). InnerVolumeSpecName "kube-api-access-5kc8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:12.055414 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.055398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-dshm" (OuterVolumeSpecName: "dshm") pod "09ff18c0-0673-4bd9-8f95-8188571be9b8" (UID: "09ff18c0-0673-4bd9-8f95-8188571be9b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:12.154555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.154507    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ff18c0-0673-4bd9-8f95-8188571be9b8-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:12.154555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.154554    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:12.154749 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.154567    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kc8d\" (UniqueName: \"kubernetes.io/projected/09ff18c0-0673-4bd9-8f95-8188571be9b8-kube-api-access-5kc8d\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:12.154749 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.154579    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ff18c0-0673-4bd9-8f95-8188571be9b8-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:12.487950 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.487864    2577 generic.go:356] "Generic (PLEG): container finished" podID="09ff18c0-0673-4bd9-8f95-8188571be9b8" containerID="064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1" exitCode=0
Sep 19 08:43:12.487950 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.487909    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"
Sep 19 08:43:12.488148 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.487953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb" event={"ID":"09ff18c0-0673-4bd9-8f95-8188571be9b8","Type":"ContainerDied","Data":"064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1"}
Sep 19 08:43:12.488148 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.488022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb" event={"ID":"09ff18c0-0673-4bd9-8f95-8188571be9b8","Type":"ContainerDied","Data":"b11157c4626bb4d96e32edc11c6cdeb8113b969982c0fd871a7fdd8c0ddacd02"}
Sep 19 08:43:12.488148 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.488047    2577 scope.go:117] "RemoveContainer" containerID="064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1"
Sep 19 08:43:12.498627 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.498610    2577 scope.go:117] "RemoveContainer" containerID="064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1"
Sep 19 08:43:12.498920 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:43:12.498901    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1\": container with ID starting with 064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1 not found: ID does not exist" containerID="064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1"
Sep 19 08:43:12.498972 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.498927    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1"} err="failed to get container status \"064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1\": rpc error: code = NotFound desc = could not find container \"064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1\": container with ID starting with 064403412ae190db15b24c5810783161cf98e98ed1f35239391a15ccd315cfc1 not found: ID does not exist"
Sep 19 08:43:12.510153 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.510121    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"]
Sep 19 08:43:12.513810 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:12.513790    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-57mwb"]
Sep 19 08:43:13.836895 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.836873    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:43:13.968848 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.968816    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-data\") pod \"c398b03f-2733-46f0-a128-e94da065630a\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") "
Sep 19 08:43:13.969015 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.968872    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c398b03f-2733-46f0-a128-e94da065630a-istio-podinfo\") pod \"c398b03f-2733-46f0-a128-e94da065630a\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") "
Sep 19 08:43:13.969015 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.968900    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c398b03f-2733-46f0-a128-e94da065630a-istiod-ca-cert\") pod \"c398b03f-2733-46f0-a128-e94da065630a\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") "
Sep 19 08:43:13.969015 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.968933    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-istio-token\") pod \"c398b03f-2733-46f0-a128-e94da065630a\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") "
Sep 19 08:43:13.969015 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.968965    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-certs\") pod \"c398b03f-2733-46f0-a128-e94da065630a\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") "
Sep 19 08:43:13.969202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969023    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvtb4\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-kube-api-access-rvtb4\") pod \"c398b03f-2733-46f0-a128-e94da065630a\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") "
Sep 19 08:43:13.969202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969062    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-credential-socket\") pod \"c398b03f-2733-46f0-a128-e94da065630a\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") "
Sep 19 08:43:13.969202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969084    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-envoy\") pod \"c398b03f-2733-46f0-a128-e94da065630a\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") "
Sep 19 08:43:13.969202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-socket\") pod \"c398b03f-2733-46f0-a128-e94da065630a\" (UID: \"c398b03f-2733-46f0-a128-e94da065630a\") "
Sep 19 08:43:13.969202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969141    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-data" (OuterVolumeSpecName: "istio-data") pod "c398b03f-2733-46f0-a128-e94da065630a" (UID: "c398b03f-2733-46f0-a128-e94da065630a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:13.969471 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969319    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c398b03f-2733-46f0-a128-e94da065630a" (UID: "c398b03f-2733-46f0-a128-e94da065630a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:13.969471 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969338    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-data\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:13.969471 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969346    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c398b03f-2733-46f0-a128-e94da065630a" (UID: "c398b03f-2733-46f0-a128-e94da065630a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:13.969586 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c398b03f-2733-46f0-a128-e94da065630a" (UID: "c398b03f-2733-46f0-a128-e94da065630a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:13.969625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.969582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c398b03f-2733-46f0-a128-e94da065630a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c398b03f-2733-46f0-a128-e94da065630a" (UID: "c398b03f-2733-46f0-a128-e94da065630a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:13.971721 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.971684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c398b03f-2733-46f0-a128-e94da065630a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c398b03f-2733-46f0-a128-e94da065630a" (UID: "c398b03f-2733-46f0-a128-e94da065630a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 08:43:13.971795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.971715    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-kube-api-access-rvtb4" (OuterVolumeSpecName: "kube-api-access-rvtb4") pod "c398b03f-2733-46f0-a128-e94da065630a" (UID: "c398b03f-2733-46f0-a128-e94da065630a"). InnerVolumeSpecName "kube-api-access-rvtb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:13.971795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.971749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c398b03f-2733-46f0-a128-e94da065630a" (UID: "c398b03f-2733-46f0-a128-e94da065630a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:13.971795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:13.971753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-istio-token" (OuterVolumeSpecName: "istio-token") pod "c398b03f-2733-46f0-a128-e94da065630a" (UID: "c398b03f-2733-46f0-a128-e94da065630a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:14.070760 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.070722    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-istio-token\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:14.070760 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.070753    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:14.070760 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.070763    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvtb4\" (UniqueName: \"kubernetes.io/projected/c398b03f-2733-46f0-a128-e94da065630a-kube-api-access-rvtb4\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:14.070989 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.070773    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-credential-socket\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:14.070989 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.070782    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-istio-envoy\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:14.070989 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.070790    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c398b03f-2733-46f0-a128-e94da065630a-workload-socket\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:14.070989 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.070798    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c398b03f-2733-46f0-a128-e94da065630a-istio-podinfo\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:14.070989 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.070806    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c398b03f-2733-46f0-a128-e94da065630a-istiod-ca-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:14.503456 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.503424    2577 generic.go:356] "Generic (PLEG): container finished" podID="c398b03f-2733-46f0-a128-e94da065630a" containerID="e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d" exitCode=0
Sep 19 08:43:14.503640 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.503535    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"
Sep 19 08:43:14.504626 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.504603    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ff18c0-0673-4bd9-8f95-8188571be9b8" path="/var/lib/kubelet/pods/09ff18c0-0673-4bd9-8f95-8188571be9b8/volumes"
Sep 19 08:43:14.504976 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.504959    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz" event={"ID":"c398b03f-2733-46f0-a128-e94da065630a","Type":"ContainerDied","Data":"e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d"}
Sep 19 08:43:14.505030 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.504993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz" event={"ID":"c398b03f-2733-46f0-a128-e94da065630a","Type":"ContainerDied","Data":"8de894c5297307816e3f5aefe5aee76890d0a86ded44dbe42743fc264311b1b2"}
Sep 19 08:43:14.505030 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.505015    2577 scope.go:117] "RemoveContainer" containerID="e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d"
Sep 19 08:43:14.516048 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.516034    2577 scope.go:117] "RemoveContainer" containerID="e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d"
Sep 19 08:43:14.516312 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:43:14.516291    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d\": container with ID starting with e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d not found: ID does not exist" containerID="e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d"
Sep 19 08:43:14.516408 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.516329    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d"} err="failed to get container status \"e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d\": rpc error: code = NotFound desc = could not find container \"e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d\": container with ID starting with e60c0d1029217cfdb1f59edca106030258dae2c80e7b4507b6dc7cb0a048261d not found: ID does not exist"
Sep 19 08:43:14.529510 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.529489    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"]
Sep 19 08:43:14.533218 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:14.533199    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-xtprz"]
Sep 19 08:43:16.503675 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:16.503646    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c398b03f-2733-46f0-a128-e94da065630a" path="/var/lib/kubelet/pods/c398b03f-2733-46f0-a128-e94da065630a/volumes"
Sep 19 08:43:51.584942 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:51.584902    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"]
Sep 19 08:43:51.585421 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:51.585141    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf" podUID="7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" containerName="istio-proxy" containerID="cri-o://61550cf06492e3c1f8b7678c933c48cea21a8e11344f0bbdab37d6dd5187f5e8" gracePeriod=30
Sep 19 08:43:53.723480 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.723447    2577 generic.go:356] "Generic (PLEG): container finished" podID="7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" containerID="61550cf06492e3c1f8b7678c933c48cea21a8e11344f0bbdab37d6dd5187f5e8" exitCode=0
Sep 19 08:43:53.723865 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.723614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf" event={"ID":"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f","Type":"ContainerDied","Data":"61550cf06492e3c1f8b7678c933c48cea21a8e11344f0bbdab37d6dd5187f5e8"}
Sep 19 08:43:53.723865 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.723671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf" event={"ID":"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f","Type":"ContainerDied","Data":"d95cdc9d30836e2ed6d0f48a8b438d50c7ce36022525c1ef57cb287e03a5de3a"}
Sep 19 08:43:53.723865 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.723689    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d95cdc9d30836e2ed6d0f48a8b438d50c7ce36022525c1ef57cb287e03a5de3a"
Sep 19 08:43:53.736852 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.736833    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:53.792548 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792522    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-token\") pod \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") "
Sep 19 08:43:53.792693 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-data\") pod \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") "
Sep 19 08:43:53.792693 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkk4n\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-kube-api-access-mkk4n\") pod \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") "
Sep 19 08:43:53.792693 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-credential-socket\") pod \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") "
Sep 19 08:43:53.792693 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792671    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istiod-ca-cert\") pod \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") "
Sep 19 08:43:53.792912 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792714    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-socket\") pod \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") "
Sep 19 08:43:53.792912 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-envoy\") pod \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") "
Sep 19 08:43:53.792912 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792806    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-podinfo\") pod \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") "
Sep 19 08:43:53.792912 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792836    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-certs\") pod \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\" (UID: \"7c17a777-2b5d-4d0f-9b82-a645a1e2d42f\") "
Sep 19 08:43:53.793111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792936    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" (UID: "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:53.793111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.792954    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" (UID: "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:53.793111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.793084    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-data\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:53.793111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.793103    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-credential-socket\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:53.793319 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.793234    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" (UID: "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:53.793522 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.793494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" (UID: "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:53.793984 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.793963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" (UID: "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:53.795387 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.795341    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" (UID: "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:53.795498 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.795382    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" (UID: "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:53.795650 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.795629    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-kube-api-access-mkk4n" (OuterVolumeSpecName: "kube-api-access-mkk4n") pod "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" (UID: "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f"). InnerVolumeSpecName "kube-api-access-mkk4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:53.795650 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.795636    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" (UID: "7c17a777-2b5d-4d0f-9b82-a645a1e2d42f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 08:43:53.893857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.893820    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkk4n\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-kube-api-access-mkk4n\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:53.893857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.893850    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istiod-ca-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:53.893857 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.893862    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-socket\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:53.894079 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.893872    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-envoy\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:53.894079 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.893884    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-podinfo\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:53.894079 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.893894    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-workload-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:53.894079 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:53.893904    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f-istio-token\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:43:54.729041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:54.729007    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"
Sep 19 08:43:54.748959 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:54.748927    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"]
Sep 19 08:43:54.752859 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:54.752834    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-xhjsf"]
Sep 19 08:43:56.504642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:43:56.504608    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" path="/var/lib/kubelet/pods/7c17a777-2b5d-4d0f-9b82-a645a1e2d42f/volumes"
Sep 19 08:44:05.717017 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.716985    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"]
Sep 19 08:44:05.717542 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.717522    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ff18c0-0673-4bd9-8f95-8188571be9b8" containerName="main"
Sep 19 08:44:05.717621 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.717544    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ff18c0-0673-4bd9-8f95-8188571be9b8" containerName="main"
Sep 19 08:44:05.717621 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.717567    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c398b03f-2733-46f0-a128-e94da065630a" containerName="istio-proxy"
Sep 19 08:44:05.717621 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.717575    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c398b03f-2733-46f0-a128-e94da065630a" containerName="istio-proxy"
Sep 19 08:44:05.717621 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.717587    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" containerName="istio-proxy"
Sep 19 08:44:05.717621 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.717596    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" containerName="istio-proxy"
Sep 19 08:44:05.717865 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.717693    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c398b03f-2733-46f0-a128-e94da065630a" containerName="istio-proxy"
Sep 19 08:44:05.717865 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.717705    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c17a777-2b5d-4d0f-9b82-a645a1e2d42f" containerName="istio-proxy"
Sep 19 08:44:05.717865 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.717717    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ff18c0-0673-4bd9-8f95-8188571be9b8" containerName="main"
Sep 19 08:44:05.722902 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.722875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.726290 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.726267    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 19 08:44:05.726418 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.726288    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 19 08:44:05.726418 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.726297    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 19 08:44:05.726418 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.726335    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-n7zqj\""
Sep 19 08:44:05.732627 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.732602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"]
Sep 19 08:44:05.788091 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.788051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.788443 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.788421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.788595 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.788580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.788719 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.788703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.788871 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.788857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.789042 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.789027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbkj\" (UniqueName: \"kubernetes.io/projected/fe6fd777-99e6-4081-93a3-6f48ee46cd62-kube-api-access-7xbkj\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.889997 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.889969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.890165 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.890008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.890165 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.890034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.890165 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.890069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.890165 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.890085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbkj\" (UniqueName: \"kubernetes.io/projected/fe6fd777-99e6-4081-93a3-6f48ee46cd62-kube-api-access-7xbkj\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.890165 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.890118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.890486 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.890444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.890558 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.890532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.890597 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.890564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.892406 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.892387    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.892578 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.892562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:05.899615 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:05.899594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbkj\" (UniqueName: \"kubernetes.io/projected/fe6fd777-99e6-4081-93a3-6f48ee46cd62-kube-api-access-7xbkj\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:06.036023 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:06.035935    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:06.310114 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:06.310085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"]
Sep 19 08:44:06.312828 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:44:06.312795    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6fd777_99e6_4081_93a3_6f48ee46cd62.slice/crio-43f2bdbf633c3112ed23dbdd02870bed09e0b0a71255e63c5895b7842f93d7d2 WatchSource:0}: Error finding container 43f2bdbf633c3112ed23dbdd02870bed09e0b0a71255e63c5895b7842f93d7d2: Status 404 returned error can't find the container with id 43f2bdbf633c3112ed23dbdd02870bed09e0b0a71255e63c5895b7842f93d7d2
Sep 19 08:44:06.796758 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:06.796723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" event={"ID":"fe6fd777-99e6-4081-93a3-6f48ee46cd62","Type":"ContainerStarted","Data":"62c86400d123759b24a62790b3b5b5e6bb5e4baa753fed4f395eddd9e4cebeb0"}
Sep 19 08:44:06.797141 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:06.796774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" event={"ID":"fe6fd777-99e6-4081-93a3-6f48ee46cd62","Type":"ContainerStarted","Data":"43f2bdbf633c3112ed23dbdd02870bed09e0b0a71255e63c5895b7842f93d7d2"}
Sep 19 08:44:06.820506 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:06.820433    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" podStartSLOduration=1.8203996789999999 podStartE2EDuration="1.820399679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:44:06.817791372 +0000 UTC m=+1414.499374805" watchObservedRunningTime="2026-09-19 08:44:06.820399679 +0000 UTC m=+1414.501983111"
Sep 19 08:44:07.036545 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:07.036506    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:16.037089 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:16.037003    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:16.049146 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:16.049124    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:16.868250 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:16.868219    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:25.472475 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.472436    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"]
Sep 19 08:44:25.476057 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.476040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.478779 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.478751    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 19 08:44:25.478915 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.478759    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-th9bg\""
Sep 19 08:44:25.478915 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.478872    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 19 08:44:25.478915 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.478763    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 19 08:44:25.479092 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.478822    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 19 08:44:25.479092 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.478761    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:44:25.487844 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.487823    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"]
Sep 19 08:44:25.577536 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.577488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-home\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.577756 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.577571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tls-certs\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.577756 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.577699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kserve-provision-location\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.577884 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.577762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-cabundle-cert\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.577884 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.577802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-model-cache\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.577884 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.577836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-dshm\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.578035 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.577886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxg2j\" (UniqueName: \"kubernetes.io/projected/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kube-api-access-fxg2j\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.578035 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.577983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tmp-dir\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.678853 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.678819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxg2j\" (UniqueName: \"kubernetes.io/projected/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kube-api-access-fxg2j\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679003 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.678861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tmp-dir\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679003 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.678903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-home\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679003 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.678936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tls-certs\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679003 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.678958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kserve-provision-location\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679003 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.678986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-cabundle-cert\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.679015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-model-cache\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679287 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.679044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-dshm\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679438 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.679409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tmp-dir\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679531 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.679507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-home\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679590 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.679572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kserve-provision-location\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679678 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.679656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-model-cache\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.679805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.679785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-cabundle-cert\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.681276 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.681251    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-dshm\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.681422 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.681406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tls-certs\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.689408 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.689390    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxg2j\" (UniqueName: \"kubernetes.io/projected/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kube-api-access-fxg2j\") pod \"auth-enabled-test-kserve-84b574c694-dpmtb\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:25.788006 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:25.787919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:44:26.065446 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:26.065419    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"]
Sep 19 08:44:26.066975 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:44:26.066945    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac6e291_bc4d_48b2_8f0a_222706ce50f3.slice/crio-465f07e8ee5440e03a7e7ceb959b6cdbb4401c4ce62f6e3b54be799dc99ee342 WatchSource:0}: Error finding container 465f07e8ee5440e03a7e7ceb959b6cdbb4401c4ce62f6e3b54be799dc99ee342: Status 404 returned error can't find the container with id 465f07e8ee5440e03a7e7ceb959b6cdbb4401c4ce62f6e3b54be799dc99ee342
Sep 19 08:44:26.911454 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:26.911403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" event={"ID":"5ac6e291-bc4d-48b2-8f0a-222706ce50f3","Type":"ContainerStarted","Data":"35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79"}
Sep 19 08:44:26.911980 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:26.911482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" event={"ID":"5ac6e291-bc4d-48b2-8f0a-222706ce50f3","Type":"ContainerStarted","Data":"465f07e8ee5440e03a7e7ceb959b6cdbb4401c4ce62f6e3b54be799dc99ee342"}
Sep 19 08:44:28.930358 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:28.930319    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerID="35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79" exitCode=0
Sep 19 08:44:28.930808 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:28.930397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" event={"ID":"5ac6e291-bc4d-48b2-8f0a-222706ce50f3","Type":"ContainerDied","Data":"35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79"}
Sep 19 08:44:28.931837 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:28.931820    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:44:52.435028 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:52.434979    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"]
Sep 19 08:44:52.447613 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:44:52.447559    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:44:52.447613 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:44:52.447613 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:44:52.447613 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="62c86400d123759b24a62790b3b5b5e6bb5e4baa753fed4f395eddd9e4cebeb0" cmd=["/bin/sleep","15"]
Sep 19 08:44:52.447933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:52.447616    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" message="" err=<
Sep 19 08:44:52.447933 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:44:52.447933 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:44:52.447933 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:44:52.447933 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:44:52.447629    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:44:52.447933 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:44:52.447933 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:44:52.447933 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" podUID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" containerName="main" containerID="cri-o://62c86400d123759b24a62790b3b5b5e6bb5e4baa753fed4f395eddd9e4cebeb0"
Sep 19 08:44:52.447933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:52.447698    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" podUID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" containerName="main" containerID="cri-o://62c86400d123759b24a62790b3b5b5e6bb5e4baa753fed4f395eddd9e4cebeb0" gracePeriod=60
Sep 19 08:44:52.857869 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:52.857827    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" podUID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 19 08:44:53.857706 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:53.857664    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" podUID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Sep 19 08:44:53.858096 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:53.857780    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:54.113696 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:54.113666    2577 generic.go:356] "Generic (PLEG): container finished" podID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" containerID="62c86400d123759b24a62790b3b5b5e6bb5e4baa753fed4f395eddd9e4cebeb0" exitCode=0
Sep 19 08:44:54.113848 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:54.113704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" event={"ID":"fe6fd777-99e6-4081-93a3-6f48ee46cd62","Type":"ContainerDied","Data":"62c86400d123759b24a62790b3b5b5e6bb5e4baa753fed4f395eddd9e4cebeb0"}
Sep 19 08:44:55.451040 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.451008    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:55.562558 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.562509    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tls-certs\") pod \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") "
Sep 19 08:44:55.562763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.562564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-home\") pod \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") "
Sep 19 08:44:55.562763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.562594    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xbkj\" (UniqueName: \"kubernetes.io/projected/fe6fd777-99e6-4081-93a3-6f48ee46cd62-kube-api-access-7xbkj\") pod \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") "
Sep 19 08:44:55.562763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.562626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-model-cache\") pod \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") "
Sep 19 08:44:55.562763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.562717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-dshm\") pod \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") "
Sep 19 08:44:55.562993 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.562790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tmp-dir\") pod \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\" (UID: \"fe6fd777-99e6-4081-93a3-6f48ee46cd62\") "
Sep 19 08:44:55.562993 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.562849    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe6fd777-99e6-4081-93a3-6f48ee46cd62" (UID: "fe6fd777-99e6-4081-93a3-6f48ee46cd62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:55.562993 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.562864    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-home" (OuterVolumeSpecName: "home") pod "fe6fd777-99e6-4081-93a3-6f48ee46cd62" (UID: "fe6fd777-99e6-4081-93a3-6f48ee46cd62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:55.563167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.563014    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe6fd777-99e6-4081-93a3-6f48ee46cd62" (UID: "fe6fd777-99e6-4081-93a3-6f48ee46cd62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:55.563167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.563098    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:44:55.563167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.563111    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:44:55.563167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.563125    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:44:55.564838 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.564809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6fd777-99e6-4081-93a3-6f48ee46cd62-kube-api-access-7xbkj" (OuterVolumeSpecName: "kube-api-access-7xbkj") pod "fe6fd777-99e6-4081-93a3-6f48ee46cd62" (UID: "fe6fd777-99e6-4081-93a3-6f48ee46cd62"). InnerVolumeSpecName "kube-api-access-7xbkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:44:55.564937 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.564892    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe6fd777-99e6-4081-93a3-6f48ee46cd62" (UID: "fe6fd777-99e6-4081-93a3-6f48ee46cd62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:44:55.565026 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.565008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-dshm" (OuterVolumeSpecName: "dshm") pod "fe6fd777-99e6-4081-93a3-6f48ee46cd62" (UID: "fe6fd777-99e6-4081-93a3-6f48ee46cd62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:55.664010 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.663974    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe6fd777-99e6-4081-93a3-6f48ee46cd62-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:44:55.664010 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.664007    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbkj\" (UniqueName: \"kubernetes.io/projected/fe6fd777-99e6-4081-93a3-6f48ee46cd62-kube-api-access-7xbkj\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:44:55.664010 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.664016    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe6fd777-99e6-4081-93a3-6f48ee46cd62-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:44:55.858543 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:55.858501    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" podUID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 19 08:44:56.129824 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:56.129719    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" event={"ID":"5ac6e291-bc4d-48b2-8f0a-222706ce50f3","Type":"ContainerStarted","Data":"e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320"}
Sep 19 08:44:56.131207 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:56.131184    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"
Sep 19 08:44:56.131348 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:56.131179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f" event={"ID":"fe6fd777-99e6-4081-93a3-6f48ee46cd62","Type":"ContainerDied","Data":"43f2bdbf633c3112ed23dbdd02870bed09e0b0a71255e63c5895b7842f93d7d2"}
Sep 19 08:44:56.131348 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:56.131311    2577 scope.go:117] "RemoveContainer" containerID="62c86400d123759b24a62790b3b5b5e6bb5e4baa753fed4f395eddd9e4cebeb0"
Sep 19 08:44:56.167922 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:56.167876    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podStartSLOduration=2.069952488 podStartE2EDuration="31.167859439s" totalImagesPullingTime="26.955732047s" totalInitContainerRuntime="2.142174904s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:44:56.163680885 +0000 UTC m=+1463.845264318" watchObservedRunningTime="2026-09-19 08:44:56.167859439 +0000 UTC m=+1463.849442917"
Sep 19 08:44:56.184133 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:56.184103    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"]
Sep 19 08:44:56.187312 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:56.187285    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-64449td4f"]
Sep 19 08:44:56.504512 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:56.504423    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" path="/var/lib/kubelet/pods/fe6fd777-99e6-4081-93a3-6f48ee46cd62/volumes"
Sep 19 08:44:56.788382 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:44:56.788283    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:45:05.788036 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:05.787996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:45:05.789342 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:05.789314    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:45:15.788523 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:15.788475    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:45:25.788779 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:25.788732    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:45:34.566958 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:34.566929    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:45:34.571972 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:34.571951    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:45:35.788349 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:35.788307    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:45:44.786114 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.786080    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"]
Sep 19 08:45:44.786655 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.786522    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" containerName="main"
Sep 19 08:45:44.786655 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.786537    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" containerName="main"
Sep 19 08:45:44.786655 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.786615    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe6fd777-99e6-4081-93a3-6f48ee46cd62" containerName="main"
Sep 19 08:45:44.795712 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.795684    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:44.798976 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.798947    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-z776q\""
Sep 19 08:45:44.799128 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.799009    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 19 08:45:44.799128 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.799009    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:45:44.799593 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.799567    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 19 08:45:44.799720 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.799703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 19 08:45:44.799847 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.799827    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 19 08:45:44.803513 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.803492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"]
Sep 19 08:45:44.908803 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.908771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:44.908803 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.908804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:44.909016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.908836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:44.909016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.908853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:44.909016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.908903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a662ef-748c-4a84-9768-296ae73d7053-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:44.909016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.908980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:44.909016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.909005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshf9\" (UniqueName: \"kubernetes.io/projected/78a662ef-748c-4a84-9768-296ae73d7053-kube-api-access-xshf9\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:44.909184 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:44.909035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a662ef-748c-4a84-9768-296ae73d7053-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.009831 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.009792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshf9\" (UniqueName: \"kubernetes.io/projected/78a662ef-748c-4a84-9768-296ae73d7053-kube-api-access-xshf9\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010027 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.009851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a662ef-748c-4a84-9768-296ae73d7053-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010027 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.009912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010027 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.009939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010027 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.009996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010027 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.010017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010311 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.010042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a662ef-748c-4a84-9768-296ae73d7053-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010311 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.010117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010545 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.010519    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010620 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.010567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.010803 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.010776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.011071 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.011045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a662ef-748c-4a84-9768-296ae73d7053-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.012467 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.012445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.013514 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.013418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.014754 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.014729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a662ef-748c-4a84-9768-296ae73d7053-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.022418 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.022352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshf9\" (UniqueName: \"kubernetes.io/projected/78a662ef-748c-4a84-9768-296ae73d7053-kube-api-access-xshf9\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-q6fpj\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.027961 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.027937    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"]
Sep 19 08:45:45.033922 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.033903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.051456 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.050015    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"]
Sep 19 08:45:45.109438 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.109398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:45.110710 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.110683    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b3398f-040d-40c3-8a02-bc3bfe3284ab-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.110807 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.110720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.110807 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.110770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.110901 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.110815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.211529 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.211491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b3398f-040d-40c3-8a02-bc3bfe3284ab-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.211529 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.211535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.211783 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.211588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.211783 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.211618    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.212043 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.212020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.212155 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.212133    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.212316 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.212296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b3398f-040d-40c3-8a02-bc3bfe3284ab-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.214237 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.214213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.354229 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.354199    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:45.391014 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.390982    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"]
Sep 19 08:45:45.393188 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:45:45.393156    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a662ef_748c_4a84_9768_296ae73d7053.slice/crio-4c550ef47ff20d107f49598cacd6b2bdd9b1fdbaa333e26bf0b51f810bde6796 WatchSource:0}: Error finding container 4c550ef47ff20d107f49598cacd6b2bdd9b1fdbaa333e26bf0b51f810bde6796: Status 404 returned error can't find the container with id 4c550ef47ff20d107f49598cacd6b2bdd9b1fdbaa333e26bf0b51f810bde6796
Sep 19 08:45:45.423108 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.423062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" event={"ID":"78a662ef-748c-4a84-9768-296ae73d7053","Type":"ContainerStarted","Data":"4c550ef47ff20d107f49598cacd6b2bdd9b1fdbaa333e26bf0b51f810bde6796"}
Sep 19 08:45:45.639104 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.639073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"]
Sep 19 08:45:45.642414 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:45:45.642380    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b3398f_040d_40c3_8a02_bc3bfe3284ab.slice/crio-12f326016c9a6e2e62d759899ccb6c6502246fdc0da06eb3f950a68772bc589a WatchSource:0}: Error finding container 12f326016c9a6e2e62d759899ccb6c6502246fdc0da06eb3f950a68772bc589a: Status 404 returned error can't find the container with id 12f326016c9a6e2e62d759899ccb6c6502246fdc0da06eb3f950a68772bc589a
Sep 19 08:45:45.788946 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:45.788897    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:45:46.434946 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:46.434899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" event={"ID":"78a662ef-748c-4a84-9768-296ae73d7053","Type":"ContainerStarted","Data":"9187f214062bf16e314f4da4e9e731dd6cdb4ab8f37bc0e9120160f515579bd3"}
Sep 19 08:45:46.437900 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:46.437864    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq" event={"ID":"96b3398f-040d-40c3-8a02-bc3bfe3284ab","Type":"ContainerStarted","Data":"c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff"}
Sep 19 08:45:46.438032 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:46.437923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq" event={"ID":"96b3398f-040d-40c3-8a02-bc3bfe3284ab","Type":"ContainerStarted","Data":"12f326016c9a6e2e62d759899ccb6c6502246fdc0da06eb3f950a68772bc589a"}
Sep 19 08:45:48.454793 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:48.454763    2577 generic.go:356] "Generic (PLEG): container finished" podID="78a662ef-748c-4a84-9768-296ae73d7053" containerID="9187f214062bf16e314f4da4e9e731dd6cdb4ab8f37bc0e9120160f515579bd3" exitCode=0
Sep 19 08:45:48.455233 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:48.454835    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" event={"ID":"78a662ef-748c-4a84-9768-296ae73d7053","Type":"ContainerDied","Data":"9187f214062bf16e314f4da4e9e731dd6cdb4ab8f37bc0e9120160f515579bd3"}
Sep 19 08:45:48.456526 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:48.456505    2577 generic.go:356] "Generic (PLEG): container finished" podID="96b3398f-040d-40c3-8a02-bc3bfe3284ab" containerID="c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff" exitCode=0
Sep 19 08:45:48.456628 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:48.456579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq" event={"ID":"96b3398f-040d-40c3-8a02-bc3bfe3284ab","Type":"ContainerDied","Data":"c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff"}
Sep 19 08:45:49.463829 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:49.463789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" event={"ID":"78a662ef-748c-4a84-9768-296ae73d7053","Type":"ContainerStarted","Data":"d6e11b3015268d7e5616baa90eeb67c111918c6597ba46119e4c43c2ab1f63e3"}
Sep 19 08:45:49.465846 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:49.465818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq" event={"ID":"96b3398f-040d-40c3-8a02-bc3bfe3284ab","Type":"ContainerStarted","Data":"b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb"}
Sep 19 08:45:49.466021 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:49.466000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:45:49.484807 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:49.484748    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" podStartSLOduration=2.877301697 podStartE2EDuration="5.484727845s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.607426148s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:45:49.483634292 +0000 UTC m=+1517.165217720" watchObservedRunningTime="2026-09-19 08:45:49.484727845 +0000 UTC m=+1517.166311278"
Sep 19 08:45:49.501762 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:49.501701    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq" podStartSLOduration=1.92765937 podStartE2EDuration="4.501680537s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.574021167s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:45:49.500154491 +0000 UTC m=+1517.181737961" watchObservedRunningTime="2026-09-19 08:45:49.501680537 +0000 UTC m=+1517.183263982"
Sep 19 08:45:50.110157 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:50.110115    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:55.109950 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:55.109899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:55.124795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:55.124763    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:55.516109 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:55.516008    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:45:55.788586 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:45:55.788477    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:46:05.788962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:05.788916    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:46:15.788307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:15.788255    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:46:20.483213 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:20.483180    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:46:21.692655 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:21.692626    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq_96b3398f-040d-40c3-8a02-bc3bfe3284ab/main/0.log"
Sep 19 08:46:22.405382 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.405211    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"]
Sep 19 08:46:22.406057 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.405585    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq" podUID="96b3398f-040d-40c3-8a02-bc3bfe3284ab" containerName="main" containerID="cri-o://b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb" gracePeriod=30
Sep 19 08:46:22.417602 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.417559    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"]
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:46:22.431920    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="d6e11b3015268d7e5616baa90eeb67c111918c6597ba46119e4c43c2ab1f63e3" cmd=["/bin/sleep","15"]
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.431991    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" message="" err=<
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:46:22.432008    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" podUID="78a662ef-748c-4a84-9768-296ae73d7053" containerName="main" containerID="cri-o://d6e11b3015268d7e5616baa90eeb67c111918c6597ba46119e4c43c2ab1f63e3"
Sep 19 08:46:22.432057 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.432053    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" podUID="78a662ef-748c-4a84-9768-296ae73d7053" containerName="main" containerID="cri-o://d6e11b3015268d7e5616baa90eeb67c111918c6597ba46119e4c43c2ab1f63e3" gracePeriod=60
Sep 19 08:46:22.501649 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.501614    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" podUID="78a662ef-748c-4a84-9768-296ae73d7053" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 19 08:46:22.671008 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.670909    2577 generic.go:356] "Generic (PLEG): container finished" podID="78a662ef-748c-4a84-9768-296ae73d7053" containerID="d6e11b3015268d7e5616baa90eeb67c111918c6597ba46119e4c43c2ab1f63e3" exitCode=0
Sep 19 08:46:22.671008 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.670966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" event={"ID":"78a662ef-748c-4a84-9768-296ae73d7053","Type":"ContainerDied","Data":"d6e11b3015268d7e5616baa90eeb67c111918c6597ba46119e4c43c2ab1f63e3"}
Sep 19 08:46:22.798066 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.798040    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:46:22.864934 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.864909    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a662ef-748c-4a84-9768-296ae73d7053-cabundle-cert\") pod \"78a662ef-748c-4a84-9768-296ae73d7053\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") "
Sep 19 08:46:22.865077 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.864943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-home\") pod \"78a662ef-748c-4a84-9768-296ae73d7053\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") "
Sep 19 08:46:22.865077 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.864986    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-dshm\") pod \"78a662ef-748c-4a84-9768-296ae73d7053\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") "
Sep 19 08:46:22.865077 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865010    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-kserve-provision-location\") pod \"78a662ef-748c-4a84-9768-296ae73d7053\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") "
Sep 19 08:46:22.865077 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865043    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xshf9\" (UniqueName: \"kubernetes.io/projected/78a662ef-748c-4a84-9768-296ae73d7053-kube-api-access-xshf9\") pod \"78a662ef-748c-4a84-9768-296ae73d7053\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") "
Sep 19 08:46:22.865299 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a662ef-748c-4a84-9768-296ae73d7053-tls-certs\") pod \"78a662ef-748c-4a84-9768-296ae73d7053\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") "
Sep 19 08:46:22.865299 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865129    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-model-cache\") pod \"78a662ef-748c-4a84-9768-296ae73d7053\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") "
Sep 19 08:46:22.865299 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865203    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-tmp-dir\") pod \"78a662ef-748c-4a84-9768-296ae73d7053\" (UID: \"78a662ef-748c-4a84-9768-296ae73d7053\") "
Sep 19 08:46:22.865299 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865271    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-home" (OuterVolumeSpecName: "home") pod "78a662ef-748c-4a84-9768-296ae73d7053" (UID: "78a662ef-748c-4a84-9768-296ae73d7053"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.865543 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865303    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a662ef-748c-4a84-9768-296ae73d7053-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "78a662ef-748c-4a84-9768-296ae73d7053" (UID: "78a662ef-748c-4a84-9768-296ae73d7053"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:46:22.865543 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865485    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a662ef-748c-4a84-9768-296ae73d7053-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.865647 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865551    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.865647 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-model-cache" (OuterVolumeSpecName: "model-cache") pod "78a662ef-748c-4a84-9768-296ae73d7053" (UID: "78a662ef-748c-4a84-9768-296ae73d7053"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.866011 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.865987    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78a662ef-748c-4a84-9768-296ae73d7053" (UID: "78a662ef-748c-4a84-9768-296ae73d7053"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.869210 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.869188    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-dshm" (OuterVolumeSpecName: "dshm") pod "78a662ef-748c-4a84-9768-296ae73d7053" (UID: "78a662ef-748c-4a84-9768-296ae73d7053"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.869872 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.869832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a662ef-748c-4a84-9768-296ae73d7053-kube-api-access-xshf9" (OuterVolumeSpecName: "kube-api-access-xshf9") pod "78a662ef-748c-4a84-9768-296ae73d7053" (UID: "78a662ef-748c-4a84-9768-296ae73d7053"). InnerVolumeSpecName "kube-api-access-xshf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:46:22.869976 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.869914    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a662ef-748c-4a84-9768-296ae73d7053-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78a662ef-748c-4a84-9768-296ae73d7053" (UID: "78a662ef-748c-4a84-9768-296ae73d7053"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:46:22.886551 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.886501    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78a662ef-748c-4a84-9768-296ae73d7053" (UID: "78a662ef-748c-4a84-9768-296ae73d7053"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.966309 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.966276    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.966309 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.966306    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.966309 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.966315    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.966595 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.966324    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a662ef-748c-4a84-9768-296ae73d7053-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.966595 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.966334    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xshf9\" (UniqueName: \"kubernetes.io/projected/78a662ef-748c-4a84-9768-296ae73d7053-kube-api-access-xshf9\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.966595 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:22.966342    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a662ef-748c-4a84-9768-296ae73d7053-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:23.056197 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.056165    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:46:23.168283 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.168246    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b3398f-040d-40c3-8a02-bc3bfe3284ab-cabundle-cert\") pod \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") "
Sep 19 08:46:23.168506 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.168315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tls-certs\") pod \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") "
Sep 19 08:46:23.168506 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.168387    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tmp-dir\") pod \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") "
Sep 19 08:46:23.168506 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.168475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-kserve-provision-location\") pod \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\" (UID: \"96b3398f-040d-40c3-8a02-bc3bfe3284ab\") "
Sep 19 08:46:23.168779 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.168750    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96b3398f-040d-40c3-8a02-bc3bfe3284ab" (UID: "96b3398f-040d-40c3-8a02-bc3bfe3284ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:23.168846 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.168698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b3398f-040d-40c3-8a02-bc3bfe3284ab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96b3398f-040d-40c3-8a02-bc3bfe3284ab" (UID: "96b3398f-040d-40c3-8a02-bc3bfe3284ab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:46:23.170813 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.170791    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96b3398f-040d-40c3-8a02-bc3bfe3284ab" (UID: "96b3398f-040d-40c3-8a02-bc3bfe3284ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:46:23.189218 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.189131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96b3398f-040d-40c3-8a02-bc3bfe3284ab" (UID: "96b3398f-040d-40c3-8a02-bc3bfe3284ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:23.269168 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.269126    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96b3398f-040d-40c3-8a02-bc3bfe3284ab-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:23.269168 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.269161    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:23.269168 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.269171    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:23.269430 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.269182    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96b3398f-040d-40c3-8a02-bc3bfe3284ab-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:46:23.678410 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.678355    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"
Sep 19 08:46:23.678410 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.678391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj" event={"ID":"78a662ef-748c-4a84-9768-296ae73d7053","Type":"ContainerDied","Data":"4c550ef47ff20d107f49598cacd6b2bdd9b1fdbaa333e26bf0b51f810bde6796"}
Sep 19 08:46:23.678658 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.678465    2577 scope.go:117] "RemoveContainer" containerID="d6e11b3015268d7e5616baa90eeb67c111918c6597ba46119e4c43c2ab1f63e3"
Sep 19 08:46:23.680290 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.680262    2577 generic.go:356] "Generic (PLEG): container finished" podID="96b3398f-040d-40c3-8a02-bc3bfe3284ab" containerID="b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb" exitCode=0
Sep 19 08:46:23.680441 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.680335    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"
Sep 19 08:46:23.680515 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.680462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq" event={"ID":"96b3398f-040d-40c3-8a02-bc3bfe3284ab","Type":"ContainerDied","Data":"b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb"}
Sep 19 08:46:23.680573 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.680520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq" event={"ID":"96b3398f-040d-40c3-8a02-bc3bfe3284ab","Type":"ContainerDied","Data":"12f326016c9a6e2e62d759899ccb6c6502246fdc0da06eb3f950a68772bc589a"}
Sep 19 08:46:23.689204 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.689167    2577 scope.go:117] "RemoveContainer" containerID="9187f214062bf16e314f4da4e9e731dd6cdb4ab8f37bc0e9120160f515579bd3"
Sep 19 08:46:23.697548 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.697531    2577 scope.go:117] "RemoveContainer" containerID="b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb"
Sep 19 08:46:23.705771 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.705737    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"]
Sep 19 08:46:23.705941 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.705905    2577 scope.go:117] "RemoveContainer" containerID="c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff"
Sep 19 08:46:23.709977 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.709954    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-q6fpj"]
Sep 19 08:46:23.715129 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.715110    2577 scope.go:117] "RemoveContainer" containerID="b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb"
Sep 19 08:46:23.715436 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:46:23.715417    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb\": container with ID starting with b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb not found: ID does not exist" containerID="b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb"
Sep 19 08:46:23.715521 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.715458    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb"} err="failed to get container status \"b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb\": rpc error: code = NotFound desc = could not find container \"b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb\": container with ID starting with b8057efc916fa68e8bc0e56f682d1cab0a0510b12d0d84b85356087ee94dd0fb not found: ID does not exist"
Sep 19 08:46:23.715521 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.715484    2577 scope.go:117] "RemoveContainer" containerID="c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff"
Sep 19 08:46:23.715758 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:46:23.715726    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff\": container with ID starting with c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff not found: ID does not exist" containerID="c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff"
Sep 19 08:46:23.715810 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.715767    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff"} err="failed to get container status \"c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff\": rpc error: code = NotFound desc = could not find container \"c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff\": container with ID starting with c82da398069b6b14a749f45450824979ea869055f356d1de395dad63fbfeecff not found: ID does not exist"
Sep 19 08:46:23.721685 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.721660    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"]
Sep 19 08:46:23.728044 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:23.728021    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-xvmqq"]
Sep 19 08:46:24.505282 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:24.505240    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a662ef-748c-4a84-9768-296ae73d7053" path="/var/lib/kubelet/pods/78a662ef-748c-4a84-9768-296ae73d7053/volumes"
Sep 19 08:46:24.505873 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:24.505852    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b3398f-040d-40c3-8a02-bc3bfe3284ab" path="/var/lib/kubelet/pods/96b3398f-040d-40c3-8a02-bc3bfe3284ab/volumes"
Sep 19 08:46:25.788808 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:25.788769    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:46:35.788643 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:35.788598    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:46:45.788540 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:45.788498    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:46:55.798644 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:55.798608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:46:55.806573 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:46:55.806519    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:47:00.687323 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:00.687287    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"]
Sep 19 08:47:15.710909 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:15.710832    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" containerID="cri-o://e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320" gracePeriod=45
Sep 19 08:47:15.799353 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:15.799311    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 19 08:47:16.463859 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.463827    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:47:16.529331 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.529252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-home\") pod \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") "
Sep 19 08:47:16.529331 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.529281    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-dshm\") pod \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") "
Sep 19 08:47:16.529331 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.529314    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tls-certs\") pod \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") "
Sep 19 08:47:16.529625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.529342    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxg2j\" (UniqueName: \"kubernetes.io/projected/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kube-api-access-fxg2j\") pod \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") "
Sep 19 08:47:16.529625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.529402    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kserve-provision-location\") pod \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") "
Sep 19 08:47:16.529625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.529460    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-model-cache\") pod \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") "
Sep 19 08:47:16.529625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.529488    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tmp-dir\") pod \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") "
Sep 19 08:47:16.529625 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.529525    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-cabundle-cert\") pod \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\" (UID: \"5ac6e291-bc4d-48b2-8f0a-222706ce50f3\") "
Sep 19 08:47:16.530010 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.529980    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-home" (OuterVolumeSpecName: "home") pod "5ac6e291-bc4d-48b2-8f0a-222706ce50f3" (UID: "5ac6e291-bc4d-48b2-8f0a-222706ce50f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:47:16.530154 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.530034    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ac6e291-bc4d-48b2-8f0a-222706ce50f3" (UID: "5ac6e291-bc4d-48b2-8f0a-222706ce50f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:47:16.530448 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.530415    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ac6e291-bc4d-48b2-8f0a-222706ce50f3" (UID: "5ac6e291-bc4d-48b2-8f0a-222706ce50f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:47:16.531914 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.531886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kube-api-access-fxg2j" (OuterVolumeSpecName: "kube-api-access-fxg2j") pod "5ac6e291-bc4d-48b2-8f0a-222706ce50f3" (UID: "5ac6e291-bc4d-48b2-8f0a-222706ce50f3"). InnerVolumeSpecName "kube-api-access-fxg2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:47:16.532039 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.532019    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-dshm" (OuterVolumeSpecName: "dshm") pod "5ac6e291-bc4d-48b2-8f0a-222706ce50f3" (UID: "5ac6e291-bc4d-48b2-8f0a-222706ce50f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:47:16.532039 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.532029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ac6e291-bc4d-48b2-8f0a-222706ce50f3" (UID: "5ac6e291-bc4d-48b2-8f0a-222706ce50f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:47:16.549833 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.549800    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ac6e291-bc4d-48b2-8f0a-222706ce50f3" (UID: "5ac6e291-bc4d-48b2-8f0a-222706ce50f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:47:16.552958 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.552936    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ac6e291-bc4d-48b2-8f0a-222706ce50f3" (UID: "5ac6e291-bc4d-48b2-8f0a-222706ce50f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:47:16.630337 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.630301    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:47:16.630337 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.630330    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:47:16.630337 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.630339    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:47:16.630337 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.630347    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxg2j\" (UniqueName: \"kubernetes.io/projected/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kube-api-access-fxg2j\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:47:16.630658 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.630356    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:47:16.630658 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.630388    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:47:16.630658 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.630397    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:47:16.630658 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.630405    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac6e291-bc4d-48b2-8f0a-222706ce50f3-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:47:16.988774 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.988738    2577 generic.go:356] "Generic (PLEG): container finished" podID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerID="e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320" exitCode=0
Sep 19 08:47:16.989158 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.988795    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"
Sep 19 08:47:16.989158 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.988817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" event={"ID":"5ac6e291-bc4d-48b2-8f0a-222706ce50f3","Type":"ContainerDied","Data":"e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320"}
Sep 19 08:47:16.989158 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.988876    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb" event={"ID":"5ac6e291-bc4d-48b2-8f0a-222706ce50f3","Type":"ContainerDied","Data":"465f07e8ee5440e03a7e7ceb959b6cdbb4401c4ce62f6e3b54be799dc99ee342"}
Sep 19 08:47:16.989158 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.988899    2577 scope.go:117] "RemoveContainer" containerID="e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320"
Sep 19 08:47:16.998351 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:16.998336    2577 scope.go:117] "RemoveContainer" containerID="35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79"
Sep 19 08:47:17.006729 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.006708    2577 scope.go:117] "RemoveContainer" containerID="e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320"
Sep 19 08:47:17.007022 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:47:17.006989    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320\": container with ID starting with e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320 not found: ID does not exist" containerID="e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320"
Sep 19 08:47:17.007121 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.007025    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320"} err="failed to get container status \"e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320\": rpc error: code = NotFound desc = could not find container \"e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320\": container with ID starting with e1d9e1baabb5242fa5220563838e92d5cf43c8edda1224e63f8ee21182e58320 not found: ID does not exist"
Sep 19 08:47:17.007121 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.007044    2577 scope.go:117] "RemoveContainer" containerID="35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79"
Sep 19 08:47:17.007321 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:47:17.007302    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79\": container with ID starting with 35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79 not found: ID does not exist" containerID="35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79"
Sep 19 08:47:17.007392 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.007328    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79"} err="failed to get container status \"35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79\": rpc error: code = NotFound desc = could not find container \"35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79\": container with ID starting with 35c32011b6d548709987f5ee0fb28b1ab18901d6f46340599ea3e9e0658f5c79 not found: ID does not exist"
Sep 19 08:47:17.013125 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.013103    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"]
Sep 19 08:47:17.016430 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.016409    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-dpmtb"]
Sep 19 08:47:17.092198 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092164    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"]
Sep 19 08:47:17.092638 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092617    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a662ef-748c-4a84-9768-296ae73d7053" containerName="storage-initializer"
Sep 19 08:47:17.092638 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092637    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a662ef-748c-4a84-9768-296ae73d7053" containerName="storage-initializer"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092651    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a662ef-748c-4a84-9768-296ae73d7053" containerName="main"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092660    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a662ef-748c-4a84-9768-296ae73d7053" containerName="main"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092669    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b3398f-040d-40c3-8a02-bc3bfe3284ab" containerName="main"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092674    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b3398f-040d-40c3-8a02-bc3bfe3284ab" containerName="main"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092686    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b3398f-040d-40c3-8a02-bc3bfe3284ab" containerName="storage-initializer"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092693    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b3398f-040d-40c3-8a02-bc3bfe3284ab" containerName="storage-initializer"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092703    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="storage-initializer"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092708    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="storage-initializer"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092718    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092723    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092781    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a662ef-748c-4a84-9768-296ae73d7053" containerName="main"
Sep 19 08:47:17.092784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092789    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b3398f-040d-40c3-8a02-bc3bfe3284ab" containerName="main"
Sep 19 08:47:17.093145 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.092795    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" containerName="main"
Sep 19 08:47:17.095979 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.095955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.098725 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.098696    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 19 08:47:17.098834 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.098727    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-769mf\""
Sep 19 08:47:17.098834 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.098746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 19 08:47:17.101550 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.101531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 19 08:47:17.101637 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.101551    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 19 08:47:17.101637 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.101533    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:47:17.111033 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.111010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"]
Sep 19 08:47:17.236555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.236521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.236755 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.236573    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.236755 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.236605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.236755 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.236667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.236755 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.236694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf27t\" (UniqueName: \"kubernetes.io/projected/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kube-api-access-nf27t\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.236755 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.236724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c8f60-54b3-4cf5-b915-e81e896759a9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.236928 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.236800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.236928 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.236838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338272 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338272 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338566 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338566 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf27t\" (UniqueName: \"kubernetes.io/projected/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kube-api-access-nf27t\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338566 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c8f60-54b3-4cf5-b915-e81e896759a9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338566 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338566 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338566 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338873 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338873 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.338983 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.338888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.339032 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.339001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.339143 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.339121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c8f60-54b3-4cf5-b915-e81e896759a9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.340715 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.340693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.341029 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.341014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.347150 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.347129    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf27t\" (UniqueName: \"kubernetes.io/projected/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kube-api-access-nf27t\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.405920 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.405878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:17.459211 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.459174    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"]
Sep 19 08:47:17.463634 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.463603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.478835 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.478758    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"]
Sep 19 08:47:17.540082 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.540048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.540246 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.540089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c63c36-8e04-4fed-90e2-834bf90c00ed-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.540246 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.540118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.540246 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.540231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c63c36-8e04-4fed-90e2-834bf90c00ed-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.640899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.640866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.640899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.640906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c63c36-8e04-4fed-90e2-834bf90c00ed-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.641159 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.640983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.641159 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.641002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c63c36-8e04-4fed-90e2-834bf90c00ed-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.641390 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.641333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.641469 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.641401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.641620 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.641600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c63c36-8e04-4fed-90e2-834bf90c00ed-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.643467 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.643446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c63c36-8e04-4fed-90e2-834bf90c00ed-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.684963 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.684936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"]
Sep 19 08:47:17.687038 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:47:17.687010    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6c8f60_54b3_4cf5_b915_e81e896759a9.slice/crio-781db9be531d1a780a4fb1c9b5ac65ca68fd1f66f006c82d88ea1c6c67b91809 WatchSource:0}: Error finding container 781db9be531d1a780a4fb1c9b5ac65ca68fd1f66f006c82d88ea1c6c67b91809: Status 404 returned error can't find the container with id 781db9be531d1a780a4fb1c9b5ac65ca68fd1f66f006c82d88ea1c6c67b91809
Sep 19 08:47:17.782219 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.782189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:17.996844 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.996745    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" event={"ID":"0e6c8f60-54b3-4cf5-b915-e81e896759a9","Type":"ContainerStarted","Data":"211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e"}
Sep 19 08:47:17.996844 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:17.996832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" event={"ID":"0e6c8f60-54b3-4cf5-b915-e81e896759a9","Type":"ContainerStarted","Data":"781db9be531d1a780a4fb1c9b5ac65ca68fd1f66f006c82d88ea1c6c67b91809"}
Sep 19 08:47:18.056786 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:18.056758    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"]
Sep 19 08:47:18.058614 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:47:18.058589    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c63c36_8e04_4fed_90e2_834bf90c00ed.slice/crio-94ea459b1b261b169fd54b24897f6f408b533fce260a2e9b8de7fe07b30252e5 WatchSource:0}: Error finding container 94ea459b1b261b169fd54b24897f6f408b533fce260a2e9b8de7fe07b30252e5: Status 404 returned error can't find the container with id 94ea459b1b261b169fd54b24897f6f408b533fce260a2e9b8de7fe07b30252e5
Sep 19 08:47:18.505493 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:18.505463    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac6e291-bc4d-48b2-8f0a-222706ce50f3" path="/var/lib/kubelet/pods/5ac6e291-bc4d-48b2-8f0a-222706ce50f3/volumes"
Sep 19 08:47:19.003508 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:19.003463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd" event={"ID":"21c63c36-8e04-4fed-90e2-834bf90c00ed","Type":"ContainerStarted","Data":"9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069"}
Sep 19 08:47:19.004047 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:19.003528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd" event={"ID":"21c63c36-8e04-4fed-90e2-834bf90c00ed","Type":"ContainerStarted","Data":"94ea459b1b261b169fd54b24897f6f408b533fce260a2e9b8de7fe07b30252e5"}
Sep 19 08:47:21.018672 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:21.018639    2577 generic.go:356] "Generic (PLEG): container finished" podID="21c63c36-8e04-4fed-90e2-834bf90c00ed" containerID="9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069" exitCode=0
Sep 19 08:47:21.019135 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:21.018733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd" event={"ID":"21c63c36-8e04-4fed-90e2-834bf90c00ed","Type":"ContainerDied","Data":"9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069"}
Sep 19 08:47:21.020343 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:21.020323    2577 generic.go:356] "Generic (PLEG): container finished" podID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" containerID="211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e" exitCode=0
Sep 19 08:47:21.020463 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:21.020376    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" event={"ID":"0e6c8f60-54b3-4cf5-b915-e81e896759a9","Type":"ContainerDied","Data":"211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e"}
Sep 19 08:47:22.028923 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:22.028892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd" event={"ID":"21c63c36-8e04-4fed-90e2-834bf90c00ed","Type":"ContainerStarted","Data":"ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009"}
Sep 19 08:47:22.029401 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:22.029113    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:47:22.030915 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:22.030891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" event={"ID":"0e6c8f60-54b3-4cf5-b915-e81e896759a9","Type":"ContainerStarted","Data":"e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec"}
Sep 19 08:47:22.049212 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:22.049166    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd" podStartSLOduration=2.605610941 podStartE2EDuration="5.049154538s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.443543597s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:47:22.046843198 +0000 UTC m=+1609.728426633" watchObservedRunningTime="2026-09-19 08:47:22.049154538 +0000 UTC m=+1609.730738023"
Sep 19 08:47:22.068296 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:22.068241    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" podStartSLOduration=2.6602968750000002 podStartE2EDuration="5.068222774s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.407925899s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:47:22.06428782 +0000 UTC m=+1609.745871252" watchObservedRunningTime="2026-09-19 08:47:22.068222774 +0000 UTC m=+1609.749806207"
Sep 19 08:47:22.406307 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:22.406269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:27.406103 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:27.406064    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:27.418905 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:27.418882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:28.077412 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:28.077378    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:47:53.049182 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:47:53.049146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:48:09.289350 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:09.289316    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd_21c63c36-8e04-4fed-90e2-834bf90c00ed/main/0.log"
Sep 19 08:48:15.595286 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:15.595252    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"]
Sep 19 08:48:15.595805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:15.595494    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd" podUID="21c63c36-8e04-4fed-90e2-834bf90c00ed" containerName="main" containerID="cri-o://ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009" gracePeriod=30
Sep 19 08:48:15.603513 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:15.603482    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"]
Sep 19 08:48:15.613972 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:48:15.613914    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:48:15.613972 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:48:15.613972 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:48:15.613972 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec" cmd=["/bin/sleep","15"]
Sep 19 08:48:15.614250 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:15.613989    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" message="" err=<
Sep 19 08:48:15.614250 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:48:15.614250 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:48:15.614250 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:48:15.614250 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:48:15.614006    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:48:15.614250 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:48:15.614250 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:48:15.614250 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" podUID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" containerName="main" containerID="cri-o://e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec"
Sep 19 08:48:15.614250 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:15.614052    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" podUID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" containerName="main" containerID="cri-o://e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec" gracePeriod=60
Sep 19 08:48:15.962702 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:15.962670    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:48:16.055193 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055162    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tls-certs\") pod \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") "
Sep 19 08:48:16.055338 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kserve-provision-location\") pod \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") "
Sep 19 08:48:16.055338 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055243    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tmp-dir\") pod \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") "
Sep 19 08:48:16.055338 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055297    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-home\") pod \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") "
Sep 19 08:48:16.055606 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055350    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c8f60-54b3-4cf5-b915-e81e896759a9-cabundle-cert\") pod \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") "
Sep 19 08:48:16.055606 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055405    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-model-cache\") pod \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") "
Sep 19 08:48:16.055606 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-dshm\") pod \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") "
Sep 19 08:48:16.055606 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055465    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf27t\" (UniqueName: \"kubernetes.io/projected/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kube-api-access-nf27t\") pod \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\" (UID: \"0e6c8f60-54b3-4cf5-b915-e81e896759a9\") "
Sep 19 08:48:16.055606 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055516    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e6c8f60-54b3-4cf5-b915-e81e896759a9" (UID: "0e6c8f60-54b3-4cf5-b915-e81e896759a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:16.055862 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055627    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-home" (OuterVolumeSpecName: "home") pod "0e6c8f60-54b3-4cf5-b915-e81e896759a9" (UID: "0e6c8f60-54b3-4cf5-b915-e81e896759a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:16.055862 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055682    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e6c8f60-54b3-4cf5-b915-e81e896759a9" (UID: "0e6c8f60-54b3-4cf5-b915-e81e896759a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:16.055862 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055739    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.055862 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055771    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.055862 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.055783    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.056119 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.056015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6c8f60-54b3-4cf5-b915-e81e896759a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e6c8f60-54b3-4cf5-b915-e81e896759a9" (UID: "0e6c8f60-54b3-4cf5-b915-e81e896759a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:48:16.058088 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.057985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e6c8f60-54b3-4cf5-b915-e81e896759a9" (UID: "0e6c8f60-54b3-4cf5-b915-e81e896759a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:48:16.058088 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.058003    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kube-api-access-nf27t" (OuterVolumeSpecName: "kube-api-access-nf27t") pod "0e6c8f60-54b3-4cf5-b915-e81e896759a9" (UID: "0e6c8f60-54b3-4cf5-b915-e81e896759a9"). InnerVolumeSpecName "kube-api-access-nf27t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:48:16.058653 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.058624    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-dshm" (OuterVolumeSpecName: "dshm") pod "0e6c8f60-54b3-4cf5-b915-e81e896759a9" (UID: "0e6c8f60-54b3-4cf5-b915-e81e896759a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:16.079283 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.079256    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e6c8f60-54b3-4cf5-b915-e81e896759a9" (UID: "0e6c8f60-54b3-4cf5-b915-e81e896759a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:16.156642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.156552    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e6c8f60-54b3-4cf5-b915-e81e896759a9-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.156642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.156579    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.156642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.156588    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf27t\" (UniqueName: \"kubernetes.io/projected/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kube-api-access-nf27t\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.156642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.156597    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e6c8f60-54b3-4cf5-b915-e81e896759a9-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.156642 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.156606    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e6c8f60-54b3-4cf5-b915-e81e896759a9-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.231272 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.231244    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:48:16.334785 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.334744    2577 generic.go:356] "Generic (PLEG): container finished" podID="21c63c36-8e04-4fed-90e2-834bf90c00ed" containerID="ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009" exitCode=0
Sep 19 08:48:16.334785 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.334791    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"
Sep 19 08:48:16.335075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.334826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd" event={"ID":"21c63c36-8e04-4fed-90e2-834bf90c00ed","Type":"ContainerDied","Data":"ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009"}
Sep 19 08:48:16.335075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.334878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd" event={"ID":"21c63c36-8e04-4fed-90e2-834bf90c00ed","Type":"ContainerDied","Data":"94ea459b1b261b169fd54b24897f6f408b533fce260a2e9b8de7fe07b30252e5"}
Sep 19 08:48:16.335075 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.334909    2577 scope.go:117] "RemoveContainer" containerID="ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009"
Sep 19 08:48:16.336888 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.336861    2577 generic.go:356] "Generic (PLEG): container finished" podID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" containerID="e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec" exitCode=0
Sep 19 08:48:16.337013 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.336914    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"
Sep 19 08:48:16.337013 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.336953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" event={"ID":"0e6c8f60-54b3-4cf5-b915-e81e896759a9","Type":"ContainerDied","Data":"e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec"}
Sep 19 08:48:16.337121 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.337010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5" event={"ID":"0e6c8f60-54b3-4cf5-b915-e81e896759a9","Type":"ContainerDied","Data":"781db9be531d1a780a4fb1c9b5ac65ca68fd1f66f006c82d88ea1c6c67b91809"}
Sep 19 08:48:16.345190 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.345165    2577 scope.go:117] "RemoveContainer" containerID="9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069"
Sep 19 08:48:16.355130 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.355108    2577 scope.go:117] "RemoveContainer" containerID="ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009"
Sep 19 08:48:16.355449 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:48:16.355422    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009\": container with ID starting with ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009 not found: ID does not exist" containerID="ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009"
Sep 19 08:48:16.355519 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.355462    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009"} err="failed to get container status \"ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009\": rpc error: code = NotFound desc = could not find container \"ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009\": container with ID starting with ab8f477e9a7698f7343e34bd711b7a8106291a0ab2c5e720b564e61dfb70b009 not found: ID does not exist"
Sep 19 08:48:16.355519 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.355487    2577 scope.go:117] "RemoveContainer" containerID="9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069"
Sep 19 08:48:16.355788 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:48:16.355767    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069\": container with ID starting with 9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069 not found: ID does not exist" containerID="9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069"
Sep 19 08:48:16.355851 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.355796    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069"} err="failed to get container status \"9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069\": rpc error: code = NotFound desc = could not find container \"9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069\": container with ID starting with 9c7ea67ac07ce064f83d14f958eacf0c5dedef586e263330904509eff90b3069 not found: ID does not exist"
Sep 19 08:48:16.355851 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.355816    2577 scope.go:117] "RemoveContainer" containerID="e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec"
Sep 19 08:48:16.358475 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.358406    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c63c36-8e04-4fed-90e2-834bf90c00ed-cabundle-cert\") pod \"21c63c36-8e04-4fed-90e2-834bf90c00ed\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") "
Sep 19 08:48:16.358475 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.358458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-tmp-dir\") pod \"21c63c36-8e04-4fed-90e2-834bf90c00ed\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") "
Sep 19 08:48:16.358654 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.358488    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c63c36-8e04-4fed-90e2-834bf90c00ed-tls-certs\") pod \"21c63c36-8e04-4fed-90e2-834bf90c00ed\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") "
Sep 19 08:48:16.358654 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.358557    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-kserve-provision-location\") pod \"21c63c36-8e04-4fed-90e2-834bf90c00ed\" (UID: \"21c63c36-8e04-4fed-90e2-834bf90c00ed\") "
Sep 19 08:48:16.359435 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.359405    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c63c36-8e04-4fed-90e2-834bf90c00ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21c63c36-8e04-4fed-90e2-834bf90c00ed" (UID: "21c63c36-8e04-4fed-90e2-834bf90c00ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:48:16.359435 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.359410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21c63c36-8e04-4fed-90e2-834bf90c00ed" (UID: "21c63c36-8e04-4fed-90e2-834bf90c00ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:16.361820 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.361799    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"]
Sep 19 08:48:16.366691 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.366667    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-g9kk5"]
Sep 19 08:48:16.367412 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.367395    2577 scope.go:117] "RemoveContainer" containerID="211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e"
Sep 19 08:48:16.369741 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.369713    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c63c36-8e04-4fed-90e2-834bf90c00ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21c63c36-8e04-4fed-90e2-834bf90c00ed" (UID: "21c63c36-8e04-4fed-90e2-834bf90c00ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:48:16.376780 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.376759    2577 scope.go:117] "RemoveContainer" containerID="e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec"
Sep 19 08:48:16.377093 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:48:16.377064    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec\": container with ID starting with e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec not found: ID does not exist" containerID="e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec"
Sep 19 08:48:16.377176 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.377097    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec"} err="failed to get container status \"e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec\": rpc error: code = NotFound desc = could not find container \"e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec\": container with ID starting with e9c56f583c6b5f0f5c5184f3c7f6cc507e5ec8b47972320cfe6a58653da707ec not found: ID does not exist"
Sep 19 08:48:16.377176 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.377114    2577 scope.go:117] "RemoveContainer" containerID="211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e"
Sep 19 08:48:16.377432 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:48:16.377408    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e\": container with ID starting with 211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e not found: ID does not exist" containerID="211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e"
Sep 19 08:48:16.377488 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.377435    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e"} err="failed to get container status \"211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e\": rpc error: code = NotFound desc = could not find container \"211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e\": container with ID starting with 211dd2cd4ad3d585fd4c9797e51d544e590c9832a099a97a8c681cdf4bd4d01e not found: ID does not exist"
Sep 19 08:48:16.386061 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.386035    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21c63c36-8e04-4fed-90e2-834bf90c00ed" (UID: "21c63c36-8e04-4fed-90e2-834bf90c00ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:16.459864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.459785    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.459864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.459812    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21c63c36-8e04-4fed-90e2-834bf90c00ed-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.459864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.459822    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21c63c36-8e04-4fed-90e2-834bf90c00ed-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.459864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.459830    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21c63c36-8e04-4fed-90e2-834bf90c00ed-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:48:16.503868 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.503838    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" path="/var/lib/kubelet/pods/0e6c8f60-54b3-4cf5-b915-e81e896759a9/volumes"
Sep 19 08:48:16.653382 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.653321    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"]
Sep 19 08:48:16.657315 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:16.657288    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-49vhd"]
Sep 19 08:48:18.504073 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:48:18.504040    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c63c36-8e04-4fed-90e2-834bf90c00ed" path="/var/lib/kubelet/pods/21c63c36-8e04-4fed-90e2-834bf90c00ed/volumes"
Sep 19 08:49:11.277432 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277392    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"]
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277757    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c63c36-8e04-4fed-90e2-834bf90c00ed" containerName="main"
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277768    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c63c36-8e04-4fed-90e2-834bf90c00ed" containerName="main"
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277788    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" containerName="storage-initializer"
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277794    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" containerName="storage-initializer"
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277802    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" containerName="main"
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277807    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" containerName="main"
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277814    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c63c36-8e04-4fed-90e2-834bf90c00ed" containerName="storage-initializer"
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277820    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c63c36-8e04-4fed-90e2-834bf90c00ed" containerName="storage-initializer"
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277869    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c63c36-8e04-4fed-90e2-834bf90c00ed" containerName="main"
Sep 19 08:49:11.279502 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.277876    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6c8f60-54b3-4cf5-b915-e81e896759a9" containerName="main"
Sep 19 08:49:11.281053 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.281034    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.283792 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.283768    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 19 08:49:11.283899 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.283802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 19 08:49:11.284170 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.284147    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 19 08:49:11.284817 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.284796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-wlb6m\""
Sep 19 08:49:11.284922 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.284796    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:49:11.284922 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.284796    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 19 08:49:11.293982 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.293961    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"]
Sep 19 08:49:11.438716 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.438680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.438716 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.438716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bb5l\" (UniqueName: \"kubernetes.io/projected/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kube-api-access-2bb5l\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.438930 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.438758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.438930 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.438790    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.438930 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.438808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.438930 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.438832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.438930 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.438917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.439099 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.438946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540403 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540403 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540403 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540708 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540708 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540708 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540708 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540708 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540563    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bb5l\" (UniqueName: \"kubernetes.io/projected/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kube-api-access-2bb5l\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540985 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540721    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"]
Sep 19 08:49:11.540985 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540753    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.540985 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.540964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.541145 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.541118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.541260 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.541213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.541326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.541283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.543227 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.543186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.543493 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.543474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.545181 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.545162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.553963 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.553940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bb5l\" (UniqueName: \"kubernetes.io/projected/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kube-api-access-2bb5l\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-m5v4l\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.555997 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.555978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"]
Sep 19 08:49:11.593326 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.593299    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:11.642166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.641700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.642166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.641755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.642166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.641782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5e9f36-11a6-417a-998d-2024c27b8b8d-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.642166 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.641888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.742838 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.742800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.742838 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.742848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.742838 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.742867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5e9f36-11a6-417a-998d-2024c27b8b8d-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.743248 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.742931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.743318 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.743277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.743390 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.743322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.743739 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.743718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5e9f36-11a6-417a-998d-2024c27b8b8d-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.745515 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.745497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-5x6td\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:11.867682 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.867657    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"]
Sep 19 08:49:11.869386 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:49:11.869343    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64f4a9e_fba1_4433_914b_f0be5f1f0e52.slice/crio-1641dca015c05316615253327c036362bac974575dcb5ae068f37b52464fb6f8 WatchSource:0}: Error finding container 1641dca015c05316615253327c036362bac974575dcb5ae068f37b52464fb6f8: Status 404 returned error can't find the container with id 1641dca015c05316615253327c036362bac974575dcb5ae068f37b52464fb6f8
Sep 19 08:49:11.878616 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:11.878594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:12.156944 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:12.156905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"]
Sep 19 08:49:12.157149 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:49:12.157126    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5e9f36_11a6_417a_998d_2024c27b8b8d.slice/crio-9870b8297345855a6ce8945663e21095b3595eb3d74d32f9084c8ad6a93e0a72 WatchSource:0}: Error finding container 9870b8297345855a6ce8945663e21095b3595eb3d74d32f9084c8ad6a93e0a72: Status 404 returned error can't find the container with id 9870b8297345855a6ce8945663e21095b3595eb3d74d32f9084c8ad6a93e0a72
Sep 19 08:49:12.641309 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:12.641270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td" event={"ID":"3e5e9f36-11a6-417a-998d-2024c27b8b8d","Type":"ContainerStarted","Data":"1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6"}
Sep 19 08:49:12.641972 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:12.641932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td" event={"ID":"3e5e9f36-11a6-417a-998d-2024c27b8b8d","Type":"ContainerStarted","Data":"9870b8297345855a6ce8945663e21095b3595eb3d74d32f9084c8ad6a93e0a72"}
Sep 19 08:49:12.643784 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:12.643760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" event={"ID":"e64f4a9e-fba1-4433-914b-f0be5f1f0e52","Type":"ContainerStarted","Data":"deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5"}
Sep 19 08:49:12.643967 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:12.643949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" event={"ID":"e64f4a9e-fba1-4433-914b-f0be5f1f0e52","Type":"ContainerStarted","Data":"1641dca015c05316615253327c036362bac974575dcb5ae068f37b52464fb6f8"}
Sep 19 08:49:14.667905 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:14.667865    2577 generic.go:356] "Generic (PLEG): container finished" podID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerID="deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5" exitCode=0
Sep 19 08:49:14.668383 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:14.667945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" event={"ID":"e64f4a9e-fba1-4433-914b-f0be5f1f0e52","Type":"ContainerDied","Data":"deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5"}
Sep 19 08:49:15.675725 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:15.675693    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e5e9f36-11a6-417a-998d-2024c27b8b8d" containerID="1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6" exitCode=0
Sep 19 08:49:15.676223 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:15.675777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td" event={"ID":"3e5e9f36-11a6-417a-998d-2024c27b8b8d","Type":"ContainerDied","Data":"1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6"}
Sep 19 08:49:15.677809 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:15.677786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" event={"ID":"e64f4a9e-fba1-4433-914b-f0be5f1f0e52","Type":"ContainerStarted","Data":"b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb"}
Sep 19 08:49:15.714079 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:15.714028    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" podStartSLOduration=2.214737827 podStartE2EDuration="4.714012785s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.499274958s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:49:15.711643396 +0000 UTC m=+1723.393226831" watchObservedRunningTime="2026-09-19 08:49:15.714012785 +0000 UTC m=+1723.395596216"
Sep 19 08:49:16.593679 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:16.593648    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:16.684975 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:16.684942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td" event={"ID":"3e5e9f36-11a6-417a-998d-2024c27b8b8d","Type":"ContainerStarted","Data":"283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e"}
Sep 19 08:49:16.685501 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:16.685235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:16.745931 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:16.745878    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td" podStartSLOduration=3.26005551 podStartE2EDuration="5.745861285s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.485805775s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:49:16.743906061 +0000 UTC m=+1724.425489493" watchObservedRunningTime="2026-09-19 08:49:16.745861285 +0000 UTC m=+1724.427444716"
Sep 19 08:49:21.594119 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:21.594082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:21.606940 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:21.606916    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:21.726011 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:21.725981    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:35.748955 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:35.748922    2577 scope.go:117] "RemoveContainer" containerID="61550cf06492e3c1f8b7678c933c48cea21a8e11344f0bbdab37d6dd5187f5e8"
Sep 19 08:49:47.704167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:47.704128    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:49.258766 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.258733    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-69cb8d959-5x6td_3e5e9f36-11a6-417a-998d-2024c27b8b8d/main/0.log"
Sep 19 08:49:49.435617 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.435580    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"]
Sep 19 08:49:49.443482 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.443448    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"]
Sep 19 08:49:49.443732 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.443684    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td" podUID="3e5e9f36-11a6-417a-998d-2024c27b8b8d" containerName="main" containerID="cri-o://283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e" gracePeriod=30
Sep 19 08:49:49.446034 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:49:49.445988    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:49:49.446034 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:49:49.446034 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:49:49.446034 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb" cmd=["/bin/sleep","15"]
Sep 19 08:49:49.446253 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.446058    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" message="" err=<
Sep 19 08:49:49.446253 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:49:49.446253 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:49:49.446253 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:49:49.446253 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:49:49.446075    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:49:49.446253 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:49:49.446253 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:49:49.446253 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" podUID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerName="main" containerID="cri-o://b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb"
Sep 19 08:49:49.446253 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.446144    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" podUID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerName="main" containerID="cri-o://b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb" gracePeriod=60
Sep 19 08:49:49.715603 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.715534    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" podUID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 19 08:49:49.805961 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.805902    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:49.876864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.876838    2577 generic.go:356] "Generic (PLEG): container finished" podID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerID="b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb" exitCode=0
Sep 19 08:49:49.876981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.876891    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"
Sep 19 08:49:49.876981 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.876924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" event={"ID":"e64f4a9e-fba1-4433-914b-f0be5f1f0e52","Type":"ContainerDied","Data":"b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb"}
Sep 19 08:49:49.877061 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.876980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l" event={"ID":"e64f4a9e-fba1-4433-914b-f0be5f1f0e52","Type":"ContainerDied","Data":"1641dca015c05316615253327c036362bac974575dcb5ae068f37b52464fb6f8"}
Sep 19 08:49:49.877061 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.877010    2577 scope.go:117] "RemoveContainer" containerID="b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb"
Sep 19 08:49:49.879575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.879554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bb5l\" (UniqueName: \"kubernetes.io/projected/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kube-api-access-2bb5l\") pod \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") "
Sep 19 08:49:49.879683 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.879620    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-cabundle-cert\") pod \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") "
Sep 19 08:49:49.879683 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.879677    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-model-cache\") pod \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") "
Sep 19 08:49:49.879757 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.879709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kserve-provision-location\") pod \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") "
Sep 19 08:49:49.879757 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.879737    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-dshm\") pod \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") "
Sep 19 08:49:49.879864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.879766    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-home\") pod \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") "
Sep 19 08:49:49.879864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.879814    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tls-certs\") pod \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") "
Sep 19 08:49:49.879864 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.879860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tmp-dir\") pod \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\" (UID: \"e64f4a9e-fba1-4433-914b-f0be5f1f0e52\") "
Sep 19 08:49:49.880124 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.880098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-home" (OuterVolumeSpecName: "home") pod "e64f4a9e-fba1-4433-914b-f0be5f1f0e52" (UID: "e64f4a9e-fba1-4433-914b-f0be5f1f0e52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:49.880207 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.880112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-model-cache" (OuterVolumeSpecName: "model-cache") pod "e64f4a9e-fba1-4433-914b-f0be5f1f0e52" (UID: "e64f4a9e-fba1-4433-914b-f0be5f1f0e52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:49.880207 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.880124    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e64f4a9e-fba1-4433-914b-f0be5f1f0e52" (UID: "e64f4a9e-fba1-4433-914b-f0be5f1f0e52"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:49:49.880320 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.880296    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e64f4a9e-fba1-4433-914b-f0be5f1f0e52" (UID: "e64f4a9e-fba1-4433-914b-f0be5f1f0e52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:49.895167 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.895137    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-dshm" (OuterVolumeSpecName: "dshm") pod "e64f4a9e-fba1-4433-914b-f0be5f1f0e52" (UID: "e64f4a9e-fba1-4433-914b-f0be5f1f0e52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:49.907788 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.907752    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e64f4a9e-fba1-4433-914b-f0be5f1f0e52" (UID: "e64f4a9e-fba1-4433-914b-f0be5f1f0e52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:49.910795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.910771    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e64f4a9e-fba1-4433-914b-f0be5f1f0e52" (UID: "e64f4a9e-fba1-4433-914b-f0be5f1f0e52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:49:49.910795 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.910787    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kube-api-access-2bb5l" (OuterVolumeSpecName: "kube-api-access-2bb5l") pod "e64f4a9e-fba1-4433-914b-f0be5f1f0e52" (UID: "e64f4a9e-fba1-4433-914b-f0be5f1f0e52"). InnerVolumeSpecName "kube-api-access-2bb5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:49:49.931707 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.931685    2577 scope.go:117] "RemoveContainer" containerID="deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5"
Sep 19 08:49:49.956948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.956916    2577 scope.go:117] "RemoveContainer" containerID="b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb"
Sep 19 08:49:49.957475 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:49:49.957450    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb\": container with ID starting with b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb not found: ID does not exist" containerID="b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb"
Sep 19 08:49:49.957592 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.957483    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb"} err="failed to get container status \"b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb\": rpc error: code = NotFound desc = could not find container \"b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb\": container with ID starting with b03e2cab5e7c352ab278da9d9475ab3a2f8903089b3035e54b01336d964c52cb not found: ID does not exist"
Sep 19 08:49:49.957592 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.957503    2577 scope.go:117] "RemoveContainer" containerID="deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5"
Sep 19 08:49:49.957805 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:49:49.957789    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5\": container with ID starting with deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5 not found: ID does not exist" containerID="deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5"
Sep 19 08:49:49.957859 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.957810    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5"} err="failed to get container status \"deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5\": rpc error: code = NotFound desc = could not find container \"deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5\": container with ID starting with deebd40b3587d3e61d5fde2726ec9a89d51a18b6a1626e6616ef70748c5ef3a5 not found: ID does not exist"
Sep 19 08:49:49.980846 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.980821    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.980846 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.980843    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bb5l\" (UniqueName: \"kubernetes.io/projected/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kube-api-access-2bb5l\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.980846 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.980851    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.981133 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.980860    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.981133 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.980868    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.981133 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.980875    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.981133 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.980884    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.981133 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:49.980893    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e64f4a9e-fba1-4433-914b-f0be5f1f0e52-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.087710 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.087684    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:50.183072 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.182709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tmp-dir\") pod \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") "
Sep 19 08:49:50.183072 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.182783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tls-certs\") pod \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") "
Sep 19 08:49:50.183072 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.182900    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-kserve-provision-location\") pod \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") "
Sep 19 08:49:50.183072 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.182972    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5e9f36-11a6-417a-998d-2024c27b8b8d-cabundle-cert\") pod \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\" (UID: \"3e5e9f36-11a6-417a-998d-2024c27b8b8d\") "
Sep 19 08:49:50.185305 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.184556    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5e9f36-11a6-417a-998d-2024c27b8b8d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e5e9f36-11a6-417a-998d-2024c27b8b8d" (UID: "3e5e9f36-11a6-417a-998d-2024c27b8b8d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:49:50.185305 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.185074    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e5e9f36-11a6-417a-998d-2024c27b8b8d" (UID: "3e5e9f36-11a6-417a-998d-2024c27b8b8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:50.185537 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.185408    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e5e9f36-11a6-417a-998d-2024c27b8b8d" (UID: "3e5e9f36-11a6-417a-998d-2024c27b8b8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:49:50.202082 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.202058    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"]
Sep 19 08:49:50.206218 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.206193    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-m5v4l"]
Sep 19 08:49:50.209315 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.209284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e5e9f36-11a6-417a-998d-2024c27b8b8d" (UID: "3e5e9f36-11a6-417a-998d-2024c27b8b8d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:50.285372 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.285346    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.285686 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.285379    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5e9f36-11a6-417a-998d-2024c27b8b8d-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.285686 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.285390    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.285686 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.285397    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e5e9f36-11a6-417a-998d-2024c27b8b8d-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.504076 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.504044    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" path="/var/lib/kubelet/pods/e64f4a9e-fba1-4433-914b-f0be5f1f0e52/volumes"
Sep 19 08:49:50.884066 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.884031    2577 generic.go:356] "Generic (PLEG): container finished" podID="3e5e9f36-11a6-417a-998d-2024c27b8b8d" containerID="283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e" exitCode=0
Sep 19 08:49:50.884237 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.884074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td" event={"ID":"3e5e9f36-11a6-417a-998d-2024c27b8b8d","Type":"ContainerDied","Data":"283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e"}
Sep 19 08:49:50.884237 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.884087    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"
Sep 19 08:49:50.884237 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.884138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td" event={"ID":"3e5e9f36-11a6-417a-998d-2024c27b8b8d","Type":"ContainerDied","Data":"9870b8297345855a6ce8945663e21095b3595eb3d74d32f9084c8ad6a93e0a72"}
Sep 19 08:49:50.884237 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.884170    2577 scope.go:117] "RemoveContainer" containerID="283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e"
Sep 19 08:49:50.892710 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.892680    2577 scope.go:117] "RemoveContainer" containerID="1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6"
Sep 19 08:49:50.900657 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.900637    2577 scope.go:117] "RemoveContainer" containerID="283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e"
Sep 19 08:49:50.900932 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:49:50.900913    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e\": container with ID starting with 283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e not found: ID does not exist" containerID="283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e"
Sep 19 08:49:50.901012 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.900945    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e"} err="failed to get container status \"283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e\": rpc error: code = NotFound desc = could not find container \"283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e\": container with ID starting with 283cdfd617396c9ce13341570a1ea487d8b3675993784f17728e03fbd25d4d7e not found: ID does not exist"
Sep 19 08:49:50.901012 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.900968    2577 scope.go:117] "RemoveContainer" containerID="1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6"
Sep 19 08:49:50.901235 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:49:50.901214    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6\": container with ID starting with 1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6 not found: ID does not exist" containerID="1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6"
Sep 19 08:49:50.901278 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.901246    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6"} err="failed to get container status \"1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6\": rpc error: code = NotFound desc = could not find container \"1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6\": container with ID starting with 1605a0e5ad3571c3f8333fbc9ea76a6466f8d3d3d09622617ccf7ed693e883e6 not found: ID does not exist"
Sep 19 08:49:50.902496 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.902481    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"]
Sep 19 08:49:50.907837 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:50.907816    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-5x6td"]
Sep 19 08:49:52.504017 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:49:52.503981    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e5e9f36-11a6-417a-998d-2024c27b8b8d" path="/var/lib/kubelet/pods/3e5e9f36-11a6-417a-998d-2024c27b8b8d/volumes"
Sep 19 08:50:34.602402 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:34.602353    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:50:34.609515 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:34.609496    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:50:49.173975 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.173934    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"]
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174284    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5e9f36-11a6-417a-998d-2024c27b8b8d" containerName="storage-initializer"
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174295    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5e9f36-11a6-417a-998d-2024c27b8b8d" containerName="storage-initializer"
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174303    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerName="main"
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174309    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerName="main"
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174333    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerName="storage-initializer"
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174340    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerName="storage-initializer"
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174346    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5e9f36-11a6-417a-998d-2024c27b8b8d" containerName="main"
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174351    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5e9f36-11a6-417a-998d-2024c27b8b8d" containerName="main"
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174421    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e64f4a9e-fba1-4433-914b-f0be5f1f0e52" containerName="main"
Sep 19 08:50:49.174575 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.174430    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e5e9f36-11a6-417a-998d-2024c27b8b8d" containerName="main"
Sep 19 08:50:49.178958 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.178938    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.182782 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.182741    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 19 08:50:49.183225 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.183203    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-74pk8\""
Sep 19 08:50:49.183322 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.183283    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 19 08:50:49.183417 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.183325    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 19 08:50:49.185724 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.185702    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"]
Sep 19 08:50:49.271519 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.271485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zm68\" (UniqueName: \"kubernetes.io/projected/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-kube-api-access-7zm68\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.271673 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.271539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.271673 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.271562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.271673 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.271593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.271673 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.271612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.271673 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.271638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.373001 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.372966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.373001 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.373004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.373233 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.373027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.373233 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.373046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.373233 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.373073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.373233 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.373110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zm68\" (UniqueName: \"kubernetes.io/projected/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-kube-api-access-7zm68\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.373587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.373563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.373587 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.373579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.373720 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.373678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.375435 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.375414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.375690 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.375670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.381250 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.381221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zm68\" (UniqueName: \"kubernetes.io/projected/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-kube-api-access-7zm68\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.405109 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.405087    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"]
Sep 19 08:50:49.408952 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.408929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:50:49.411345 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.411306    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-rmqqx\""
Sep 19 08:50:49.415509 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.415486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"]
Sep 19 08:50:49.473954 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.473896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcw54\" (UniqueName: \"kubernetes.io/projected/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-kube-api-access-bcw54\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2\" (UID: \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:50:49.474052 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.474000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2\" (UID: \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:50:49.490897 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.490875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:49.575250 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.575212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcw54\" (UniqueName: \"kubernetes.io/projected/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-kube-api-access-bcw54\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2\" (UID: \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:50:49.575408 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.575387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2\" (UID: \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:50:49.577902 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.577872    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2\" (UID: \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:50:49.582848 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.582823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcw54\" (UniqueName: \"kubernetes.io/projected/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-kube-api-access-bcw54\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2\" (UID: \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:50:49.720837 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.720804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:50:49.764667 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.764638    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"]
Sep 19 08:50:49.767034 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:50:49.766996    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ead8ab_bdbd_4794_8449_0e6a529c9cc9.slice/crio-37b24a8b13bef133417d2f1e18976a8dd533b2ec6eb98b9ebc48baad64c68bac WatchSource:0}: Error finding container 37b24a8b13bef133417d2f1e18976a8dd533b2ec6eb98b9ebc48baad64c68bac: Status 404 returned error can't find the container with id 37b24a8b13bef133417d2f1e18976a8dd533b2ec6eb98b9ebc48baad64c68bac
Sep 19 08:50:49.991678 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:49.991649    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"]
Sep 19 08:50:49.993271 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:50:49.993246    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7843c4c_c2fe_4a8d_ac9e_d9814c6bea35.slice/crio-6300a3417f52b7b9b951b551b4033bb565905e41e741e157508953e54b232ddf WatchSource:0}: Error finding container 6300a3417f52b7b9b951b551b4033bb565905e41e741e157508953e54b232ddf: Status 404 returned error can't find the container with id 6300a3417f52b7b9b951b551b4033bb565905e41e741e157508953e54b232ddf
Sep 19 08:50:50.198700 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:50.198664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" event={"ID":"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35","Type":"ContainerStarted","Data":"6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146"}
Sep 19 08:50:50.199169 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:50.198721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" event={"ID":"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35","Type":"ContainerStarted","Data":"6300a3417f52b7b9b951b551b4033bb565905e41e741e157508953e54b232ddf"}
Sep 19 08:50:50.199169 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:50.198764    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:50:50.200258 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:50.200232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x" event={"ID":"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9","Type":"ContainerStarted","Data":"43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32"}
Sep 19 08:50:50.200405 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:50.200274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x" event={"ID":"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9","Type":"ContainerStarted","Data":"37b24a8b13bef133417d2f1e18976a8dd533b2ec6eb98b9ebc48baad64c68bac"}
Sep 19 08:50:50.215237 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:50.215190    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" podStartSLOduration=1.215177554 podStartE2EDuration="1.215177554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:50:50.214013887 +0000 UTC m=+1817.895597322" watchObservedRunningTime="2026-09-19 08:50:50.215177554 +0000 UTC m=+1817.896760986"
Sep 19 08:50:50.231931 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:50.231889    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x" podStartSLOduration=1.23187543 podStartE2EDuration="1.23187543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:50:50.229840516 +0000 UTC m=+1817.911423948" watchObservedRunningTime="2026-09-19 08:50:50.23187543 +0000 UTC m=+1817.913458861"
Sep 19 08:50:50.491448 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:50.491417    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:59.491992 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:59.491956    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:50:59.504685 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:50:59.504656    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:51:00.265475 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:00.265441    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:51:21.207172 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:21.207141    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:51:22.620411 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:22.620261    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"]
Sep 19 08:51:22.623988 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:22.623960    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"]
Sep 19 08:51:22.635708 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:51:22.635662    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:51:22.635708 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:51:22.635708 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:51:22.635708 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32" cmd=["/bin/sleep","15"]
Sep 19 08:51:22.636044 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:22.635722    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x" message="" err=<
Sep 19 08:51:22.636044 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:51:22.636044 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:51:22.636044 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:51:22.636044 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:51:22.635736    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:51:22.636044 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:51:22.636044 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:51:22.636044 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x" podUID="f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" containerName="main" containerID="cri-o://43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32"
Sep 19 08:51:22.636044 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:22.635770    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x" podUID="f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" containerName="main" containerID="cri-o://43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32" gracePeriod=60
Sep 19 08:51:22.983862 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:22.983835    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:51:23.066555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.066519    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-home\") pod \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") "
Sep 19 08:51:23.066753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.066568    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tls-certs\") pod \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") "
Sep 19 08:51:23.066753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.066592    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tmp-dir\") pod \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") "
Sep 19 08:51:23.066753 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.066616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-dshm\") pod \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") "
Sep 19 08:51:23.066924 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.066765    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zm68\" (UniqueName: \"kubernetes.io/projected/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-kube-api-access-7zm68\") pod \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") "
Sep 19 08:51:23.066924 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.066836    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-model-cache\") pod \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\" (UID: \"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9\") "
Sep 19 08:51:23.066924 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.066847    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-home" (OuterVolumeSpecName: "home") pod "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" (UID: "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:51:23.067041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.066920    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" (UID: "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:51:23.067079 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.067042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" (UID: "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:51:23.067163 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.067149    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:51:23.067202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.067167    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:51:23.067202 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.067181    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:51:23.068926 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.068900    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-kube-api-access-7zm68" (OuterVolumeSpecName: "kube-api-access-7zm68") pod "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" (UID: "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9"). InnerVolumeSpecName "kube-api-access-7zm68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:51:23.069225 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.069201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-dshm" (OuterVolumeSpecName: "dshm") pod "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" (UID: "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:51:23.069225 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.069207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" (UID: "f8ead8ab-bdbd-4794-8449-0e6a529c9cc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:51:23.168203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.168114    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zm68\" (UniqueName: \"kubernetes.io/projected/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-kube-api-access-7zm68\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:51:23.168203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.168146    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:51:23.168203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.168157    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:51:23.379021 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.378986    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" containerID="43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32" exitCode=0
Sep 19 08:51:23.379193 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.379034    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"
Sep 19 08:51:23.379193 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.379071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x" event={"ID":"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9","Type":"ContainerDied","Data":"43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32"}
Sep 19 08:51:23.379193 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.379141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x" event={"ID":"f8ead8ab-bdbd-4794-8449-0e6a529c9cc9","Type":"ContainerDied","Data":"37b24a8b13bef133417d2f1e18976a8dd533b2ec6eb98b9ebc48baad64c68bac"}
Sep 19 08:51:23.379193 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.379164    2577 scope.go:117] "RemoveContainer" containerID="43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32"
Sep 19 08:51:23.389222 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.389207    2577 scope.go:117] "RemoveContainer" containerID="43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32"
Sep 19 08:51:23.389563 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:51:23.389544    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32\": container with ID starting with 43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32 not found: ID does not exist" containerID="43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32"
Sep 19 08:51:23.389648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.389569    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32"} err="failed to get container status \"43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32\": rpc error: code = NotFound desc = could not find container \"43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32\": container with ID starting with 43665244ca20ab362a074b23ecbf1c2249f2faa8ae748cd57c72e1be26b47a32 not found: ID does not exist"
Sep 19 08:51:23.400299 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.400273    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"]
Sep 19 08:51:23.403747 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:23.403727    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-ltd9x"]
Sep 19 08:51:24.504005 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:24.503973    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" path="/var/lib/kubelet/pods/f8ead8ab-bdbd-4794-8449-0e6a529c9cc9/volumes"
Sep 19 08:51:31.206831 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:31.206782    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" podUID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:51:37.644688 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:37.644611    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" podUID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" containerName="main" containerID="cri-o://6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146" gracePeriod=45
Sep 19 08:51:41.207517 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:41.207470    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" podUID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:51:51.207484 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:51.207358    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" podUID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:51:51.207868 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:51:51.207503    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:52:01.206797 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:01.206755    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" podUID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:52:07.895208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:07.895175    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:52:07.955901 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:07.955816    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-tls-certs\") pod \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\" (UID: \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\") "
Sep 19 08:52:07.955901 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:07.955874    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcw54\" (UniqueName: \"kubernetes.io/projected/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-kube-api-access-bcw54\") pod \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\" (UID: \"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35\") "
Sep 19 08:52:07.957932 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:07.957887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" (UID: "d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:52:07.958048 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:07.958031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-kube-api-access-bcw54" (OuterVolumeSpecName: "kube-api-access-bcw54") pod "d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" (UID: "d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35"). InnerVolumeSpecName "kube-api-access-bcw54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:52:08.057106 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.057060    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcw54\" (UniqueName: \"kubernetes.io/projected/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-kube-api-access-bcw54\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:52:08.057106 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.057101    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:52:08.619793 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.619757    2577 generic.go:356] "Generic (PLEG): container finished" podID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" containerID="6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146" exitCode=0
Sep 19 08:52:08.619962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.619815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" event={"ID":"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35","Type":"ContainerDied","Data":"6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146"}
Sep 19 08:52:08.619962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.619852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2" event={"ID":"d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35","Type":"ContainerDied","Data":"6300a3417f52b7b9b951b551b4033bb565905e41e741e157508953e54b232ddf"}
Sep 19 08:52:08.619962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.619872    2577 scope.go:117] "RemoveContainer" containerID="6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146"
Sep 19 08:52:08.619962 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.619816    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"
Sep 19 08:52:08.629176 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.629152    2577 scope.go:117] "RemoveContainer" containerID="6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146"
Sep 19 08:52:08.629478 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:52:08.629451    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146\": container with ID starting with 6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146 not found: ID does not exist" containerID="6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146"
Sep 19 08:52:08.629536 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.629481    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146"} err="failed to get container status \"6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146\": rpc error: code = NotFound desc = could not find container \"6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146\": container with ID starting with 6208a443fe0c97e39594f069003874c29a3c0bdabf70eb4910fbf18bdeb3f146 not found: ID does not exist"
Sep 19 08:52:08.637555 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.637532    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"]
Sep 19 08:52:08.639680 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:08.639659    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-sche27jz2"]
Sep 19 08:52:10.503773 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:10.503738    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" path="/var/lib/kubelet/pods/d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35/volumes"
Sep 19 08:52:27.073667 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.073625    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"]
Sep 19 08:52:27.074246 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.074222    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" containerName="main"
Sep 19 08:52:27.074398 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.074316    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" containerName="main"
Sep 19 08:52:27.074398 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.074380    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" containerName="main"
Sep 19 08:52:27.074398 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.074389    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" containerName="main"
Sep 19 08:52:27.074604 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.074500    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7843c4c-c2fe-4a8d-ac9e-d9814c6bea35" containerName="main"
Sep 19 08:52:27.074604 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.074521    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ead8ab-bdbd-4794-8449-0e6a529c9cc9" containerName="main"
Sep 19 08:52:27.078880 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.078857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.082419 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.082395    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 19 08:52:27.082419 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.082395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 19 08:52:27.082630 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.082402    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 19 08:52:27.082630 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.082402    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-bslbm\""
Sep 19 08:52:27.089040 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.089011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"]
Sep 19 08:52:27.122473 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.122439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s45s\" (UniqueName: \"kubernetes.io/projected/70fc413e-37bc-45df-b14a-21df17ea52fd-kube-api-access-9s45s\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.122473 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.122476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.122721 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.122546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.122721 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.122581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.122721 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.122655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.122721 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.122678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70fc413e-37bc-45df-b14a-21df17ea52fd-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.224174 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.224129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.224174 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.224177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70fc413e-37bc-45df-b14a-21df17ea52fd-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.224487 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.224212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s45s\" (UniqueName: \"kubernetes.io/projected/70fc413e-37bc-45df-b14a-21df17ea52fd-kube-api-access-9s45s\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.224487 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.224231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.224487 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.224285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.224487 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.224305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.224702 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.224647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.224702 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.224679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.224768 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.224705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.226581 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.226539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.227064 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.227041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70fc413e-37bc-45df-b14a-21df17ea52fd-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.232709 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.232691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s45s\" (UniqueName: \"kubernetes.io/projected/70fc413e-37bc-45df-b14a-21df17ea52fd-kube-api-access-9s45s\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.268437 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.268396    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"]
Sep 19 08:52:27.272035 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.272015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:52:27.274546 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.274523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-zlqpq\""
Sep 19 08:52:27.280863 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.280798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"]
Sep 19 08:52:27.325130 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.325043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9856df54-34ac-4784-99d7-beb7d90811f4-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph\" (UID: \"9856df54-34ac-4784-99d7-beb7d90811f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:52:27.325130 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.325079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w4qk\" (UniqueName: \"kubernetes.io/projected/9856df54-34ac-4784-99d7-beb7d90811f4-kube-api-access-6w4qk\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph\" (UID: \"9856df54-34ac-4784-99d7-beb7d90811f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:52:27.390191 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.390147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:27.426103 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.426060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9856df54-34ac-4784-99d7-beb7d90811f4-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph\" (UID: \"9856df54-34ac-4784-99d7-beb7d90811f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:52:27.426103 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.426103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w4qk\" (UniqueName: \"kubernetes.io/projected/9856df54-34ac-4784-99d7-beb7d90811f4-kube-api-access-6w4qk\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph\" (UID: \"9856df54-34ac-4784-99d7-beb7d90811f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:52:27.428907 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.428848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9856df54-34ac-4784-99d7-beb7d90811f4-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph\" (UID: \"9856df54-34ac-4784-99d7-beb7d90811f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:52:27.434401 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.434349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w4qk\" (UniqueName: \"kubernetes.io/projected/9856df54-34ac-4784-99d7-beb7d90811f4-kube-api-access-6w4qk\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph\" (UID: \"9856df54-34ac-4784-99d7-beb7d90811f4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:52:27.584752 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.584659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:52:27.664427 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.664391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"]
Sep 19 08:52:27.666721 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:52:27.666691    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fc413e_37bc_45df_b14a_21df17ea52fd.slice/crio-b2b4beaa8a7e57bb84fcd20d7b961b551d2c656d3b5de05874f3cab8ace26fd9 WatchSource:0}: Error finding container b2b4beaa8a7e57bb84fcd20d7b961b551d2c656d3b5de05874f3cab8ace26fd9: Status 404 returned error can't find the container with id b2b4beaa8a7e57bb84fcd20d7b961b551d2c656d3b5de05874f3cab8ace26fd9
Sep 19 08:52:27.728351 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.728315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s" event={"ID":"70fc413e-37bc-45df-b14a-21df17ea52fd","Type":"ContainerStarted","Data":"b2b4beaa8a7e57bb84fcd20d7b961b551d2c656d3b5de05874f3cab8ace26fd9"}
Sep 19 08:52:27.858992 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:27.858964    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"]
Sep 19 08:52:27.860425 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:52:27.860398    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9856df54_34ac_4784_99d7_beb7d90811f4.slice/crio-99434d1af3b5c000e4c11ce326a0564c719c87f94eb548abf1f698e9c8c1944b WatchSource:0}: Error finding container 99434d1af3b5c000e4c11ce326a0564c719c87f94eb548abf1f698e9c8c1944b: Status 404 returned error can't find the container with id 99434d1af3b5c000e4c11ce326a0564c719c87f94eb548abf1f698e9c8c1944b
Sep 19 08:52:28.735355 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:28.735315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s" event={"ID":"70fc413e-37bc-45df-b14a-21df17ea52fd","Type":"ContainerStarted","Data":"3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e"}
Sep 19 08:52:28.736915 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:28.736888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" event={"ID":"9856df54-34ac-4784-99d7-beb7d90811f4","Type":"ContainerStarted","Data":"b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325"}
Sep 19 08:52:28.737038 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:28.736939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" event={"ID":"9856df54-34ac-4784-99d7-beb7d90811f4","Type":"ContainerStarted","Data":"99434d1af3b5c000e4c11ce326a0564c719c87f94eb548abf1f698e9c8c1944b"}
Sep 19 08:52:28.737038 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:28.736969    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:52:28.754645 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:28.754592    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s" podStartSLOduration=1.7545751219999999 podStartE2EDuration="1.754575122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:52:28.754240131 +0000 UTC m=+1916.435823576" watchObservedRunningTime="2026-09-19 08:52:28.754575122 +0000 UTC m=+1916.436158555"
Sep 19 08:52:28.771980 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:28.771925    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" podStartSLOduration=1.771910908 podStartE2EDuration="1.771910908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:52:28.771533394 +0000 UTC m=+1916.453116830" watchObservedRunningTime="2026-09-19 08:52:28.771910908 +0000 UTC m=+1916.453494351"
Sep 19 08:52:29.390463 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:29.390407    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:37.390504 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:37.390449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:37.402876 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:37.402847    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:37.796741 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:37.796656    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:52:59.744863 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:52:59.744826    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:53:11.165568 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.165527    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"]
Sep 19 08:53:11.169709 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.169683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.172209 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.172184    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 19 08:53:11.177852 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.177823    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"]
Sep 19 08:53:11.224049 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.224017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.224269 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.224057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52efe57b-6c93-4f91-a1ac-74b2153842a7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.224269 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.224086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.224269 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.224168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.224269 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.224255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.224500 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.224293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zm8\" (UniqueName: \"kubernetes.io/projected/52efe57b-6c93-4f91-a1ac-74b2153842a7-kube-api-access-n7zm8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.324905 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.324863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zm8\" (UniqueName: \"kubernetes.io/projected/52efe57b-6c93-4f91-a1ac-74b2153842a7-kube-api-access-n7zm8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.325111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.324913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.325111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.325034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52efe57b-6c93-4f91-a1ac-74b2153842a7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.325111 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.325096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.325308 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.325153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.325308 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.325237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.325464 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.325425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.325582 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.325556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.325648 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.325580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.327335 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.327310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.327619 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.327576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52efe57b-6c93-4f91-a1ac-74b2153842a7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.332761 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.332738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zm8\" (UniqueName: \"kubernetes.io/projected/52efe57b-6c93-4f91-a1ac-74b2153842a7-kube-api-access-n7zm8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.481209 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.481118    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:11.755039 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.755010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"]
Sep 19 08:53:11.757597 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:53:11.757563    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52efe57b_6c93_4f91_a1ac_74b2153842a7.slice/crio-d36019d2d203898f75e0134ca85f65deba90367dddae7d16230a409f4e08cc64 WatchSource:0}: Error finding container d36019d2d203898f75e0134ca85f65deba90367dddae7d16230a409f4e08cc64: Status 404 returned error can't find the container with id d36019d2d203898f75e0134ca85f65deba90367dddae7d16230a409f4e08cc64
Sep 19 08:53:11.971212 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.971167    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t" event={"ID":"52efe57b-6c93-4f91-a1ac-74b2153842a7","Type":"ContainerStarted","Data":"bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594"}
Sep 19 08:53:11.971212 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.971218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t" event={"ID":"52efe57b-6c93-4f91-a1ac-74b2153842a7","Type":"ContainerStarted","Data":"d36019d2d203898f75e0134ca85f65deba90367dddae7d16230a409f4e08cc64"}
Sep 19 08:53:11.990716 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:11.990658    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t" podStartSLOduration=0.990644806 podStartE2EDuration="990.644806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:53:11.989969055 +0000 UTC m=+1959.671552488" watchObservedRunningTime="2026-09-19 08:53:11.990644806 +0000 UTC m=+1959.672228238"
Sep 19 08:53:12.481537 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:12.481500    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:21.482012 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:21.481926    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:21.494720 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:21.494693    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:22.035581 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:22.035551    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:47.315458 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.315409    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"]
Sep 19 08:53:47.325663 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:53:47.325622    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:53:47.325663 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:53:47.325663 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:53:47.325663 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594" cmd=["/bin/sleep","15"]
Sep 19 08:53:47.326016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.325679    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t" message="" err=<
Sep 19 08:53:47.326016 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:53:47.326016 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:53:47.326016 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:53:47.326016 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:53:47.325692    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:53:47.326016 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:53:47.326016 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:53:47.326016 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t" podUID="52efe57b-6c93-4f91-a1ac-74b2153842a7" containerName="main" containerID="cri-o://bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594"
Sep 19 08:53:47.326016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.325742    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t" podUID="52efe57b-6c93-4f91-a1ac-74b2153842a7" containerName="main" containerID="cri-o://bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594" gracePeriod=60
Sep 19 08:53:47.679448 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.679420    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:47.751487 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.751454    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52efe57b-6c93-4f91-a1ac-74b2153842a7-tls-certs\") pod \"52efe57b-6c93-4f91-a1ac-74b2153842a7\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") "
Sep 19 08:53:47.751671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.751517    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-model-cache\") pod \"52efe57b-6c93-4f91-a1ac-74b2153842a7\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") "
Sep 19 08:53:47.751671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.751549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-tmp-dir\") pod \"52efe57b-6c93-4f91-a1ac-74b2153842a7\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") "
Sep 19 08:53:47.751671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.751574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7zm8\" (UniqueName: \"kubernetes.io/projected/52efe57b-6c93-4f91-a1ac-74b2153842a7-kube-api-access-n7zm8\") pod \"52efe57b-6c93-4f91-a1ac-74b2153842a7\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") "
Sep 19 08:53:47.751671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.751609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-dshm\") pod \"52efe57b-6c93-4f91-a1ac-74b2153842a7\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") "
Sep 19 08:53:47.751671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.751645    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-home\") pod \"52efe57b-6c93-4f91-a1ac-74b2153842a7\" (UID: \"52efe57b-6c93-4f91-a1ac-74b2153842a7\") "
Sep 19 08:53:47.751974 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.751822    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "52efe57b-6c93-4f91-a1ac-74b2153842a7" (UID: "52efe57b-6c93-4f91-a1ac-74b2153842a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:47.751974 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.751917    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:53:47.752085 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.752019    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-home" (OuterVolumeSpecName: "home") pod "52efe57b-6c93-4f91-a1ac-74b2153842a7" (UID: "52efe57b-6c93-4f91-a1ac-74b2153842a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:47.752138 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.752071    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52efe57b-6c93-4f91-a1ac-74b2153842a7" (UID: "52efe57b-6c93-4f91-a1ac-74b2153842a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:47.753971 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.753948    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52efe57b-6c93-4f91-a1ac-74b2153842a7-kube-api-access-n7zm8" (OuterVolumeSpecName: "kube-api-access-n7zm8") pod "52efe57b-6c93-4f91-a1ac-74b2153842a7" (UID: "52efe57b-6c93-4f91-a1ac-74b2153842a7"). InnerVolumeSpecName "kube-api-access-n7zm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:53:47.754090 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.754070    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52efe57b-6c93-4f91-a1ac-74b2153842a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52efe57b-6c93-4f91-a1ac-74b2153842a7" (UID: "52efe57b-6c93-4f91-a1ac-74b2153842a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:53:47.754161 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.754085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-dshm" (OuterVolumeSpecName: "dshm") pod "52efe57b-6c93-4f91-a1ac-74b2153842a7" (UID: "52efe57b-6c93-4f91-a1ac-74b2153842a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:47.852898 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.852863    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52efe57b-6c93-4f91-a1ac-74b2153842a7-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:53:47.852898 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.852897    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:53:47.853109 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.852907    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zm8\" (UniqueName: \"kubernetes.io/projected/52efe57b-6c93-4f91-a1ac-74b2153842a7-kube-api-access-n7zm8\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:53:47.853109 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.852916    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:53:47.853109 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:47.852925    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52efe57b-6c93-4f91-a1ac-74b2153842a7-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:53:48.173989 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.173902    2577 generic.go:356] "Generic (PLEG): container finished" podID="52efe57b-6c93-4f91-a1ac-74b2153842a7" containerID="bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594" exitCode=0
Sep 19 08:53:48.173989 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.173950    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"
Sep 19 08:53:48.173989 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.173964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t" event={"ID":"52efe57b-6c93-4f91-a1ac-74b2153842a7","Type":"ContainerDied","Data":"bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594"}
Sep 19 08:53:48.174244 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.174005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t" event={"ID":"52efe57b-6c93-4f91-a1ac-74b2153842a7","Type":"ContainerDied","Data":"d36019d2d203898f75e0134ca85f65deba90367dddae7d16230a409f4e08cc64"}
Sep 19 08:53:48.174244 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.174028    2577 scope.go:117] "RemoveContainer" containerID="bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594"
Sep 19 08:53:48.184444 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.184420    2577 scope.go:117] "RemoveContainer" containerID="bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594"
Sep 19 08:53:48.184760 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:53:48.184740    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594\": container with ID starting with bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594 not found: ID does not exist" containerID="bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594"
Sep 19 08:53:48.184810 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.184770    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594"} err="failed to get container status \"bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594\": rpc error: code = NotFound desc = could not find container \"bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594\": container with ID starting with bebd691d397299da8d40d1f683248641d0fa18fcef7e3948879d9546cd9a9594 not found: ID does not exist"
Sep 19 08:53:48.196990 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.196958    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"]
Sep 19 08:53:48.200190 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.200164    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995lwp5t"]
Sep 19 08:53:48.503926 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:53:48.503846    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52efe57b-6c93-4f91-a1ac-74b2153842a7" path="/var/lib/kubelet/pods/52efe57b-6c93-4f91-a1ac-74b2153842a7/volumes"
Sep 19 08:54:49.001374 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.001277    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"]
Sep 19 08:54:49.006683 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.006651    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"]
Sep 19 08:54:49.011913 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:54:49.011873    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:54:49.011913 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:54:49.011913 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:54:49.011913 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e" cmd=["/bin/sleep","15"]
Sep 19 08:54:49.012201 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.011935    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s" message="" err=<
Sep 19 08:54:49.012201 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:54:49.012201 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:54:49.012201 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:54:49.012201 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:54:49.011953    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:54:49.012201 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:54:49.012201 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:54:49.012201 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s" podUID="70fc413e-37bc-45df-b14a-21df17ea52fd" containerName="main" containerID="cri-o://3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e"
Sep 19 08:54:49.012201 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.012007    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s" podUID="70fc413e-37bc-45df-b14a-21df17ea52fd" containerName="main" containerID="cri-o://3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e" gracePeriod=60
Sep 19 08:54:49.374131 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.374102    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:54:49.482299 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482267    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-model-cache\") pod \"70fc413e-37bc-45df-b14a-21df17ea52fd\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") "
Sep 19 08:54:49.482490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-tmp-dir\") pod \"70fc413e-37bc-45df-b14a-21df17ea52fd\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") "
Sep 19 08:54:49.482490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-dshm\") pod \"70fc413e-37bc-45df-b14a-21df17ea52fd\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") "
Sep 19 08:54:49.482490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482402    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70fc413e-37bc-45df-b14a-21df17ea52fd-tls-certs\") pod \"70fc413e-37bc-45df-b14a-21df17ea52fd\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") "
Sep 19 08:54:49.482490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482439    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s45s\" (UniqueName: \"kubernetes.io/projected/70fc413e-37bc-45df-b14a-21df17ea52fd-kube-api-access-9s45s\") pod \"70fc413e-37bc-45df-b14a-21df17ea52fd\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") "
Sep 19 08:54:49.482490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-home\") pod \"70fc413e-37bc-45df-b14a-21df17ea52fd\" (UID: \"70fc413e-37bc-45df-b14a-21df17ea52fd\") "
Sep 19 08:54:49.482794 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482611    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "70fc413e-37bc-45df-b14a-21df17ea52fd" (UID: "70fc413e-37bc-45df-b14a-21df17ea52fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:54:49.482794 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70fc413e-37bc-45df-b14a-21df17ea52fd" (UID: "70fc413e-37bc-45df-b14a-21df17ea52fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:54:49.482911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482805    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:54:49.482911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482824    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:54:49.482911 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.482860    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-home" (OuterVolumeSpecName: "home") pod "70fc413e-37bc-45df-b14a-21df17ea52fd" (UID: "70fc413e-37bc-45df-b14a-21df17ea52fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:54:49.485415 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.485388    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70fc413e-37bc-45df-b14a-21df17ea52fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70fc413e-37bc-45df-b14a-21df17ea52fd" (UID: "70fc413e-37bc-45df-b14a-21df17ea52fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:54:49.485552 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.485531    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fc413e-37bc-45df-b14a-21df17ea52fd-kube-api-access-9s45s" (OuterVolumeSpecName: "kube-api-access-9s45s") pod "70fc413e-37bc-45df-b14a-21df17ea52fd" (UID: "70fc413e-37bc-45df-b14a-21df17ea52fd"). InnerVolumeSpecName "kube-api-access-9s45s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:54:49.485621 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.485555    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-dshm" (OuterVolumeSpecName: "dshm") pod "70fc413e-37bc-45df-b14a-21df17ea52fd" (UID: "70fc413e-37bc-45df-b14a-21df17ea52fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:54:49.507194 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.507163    2577 generic.go:356] "Generic (PLEG): container finished" podID="70fc413e-37bc-45df-b14a-21df17ea52fd" containerID="3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e" exitCode=0
Sep 19 08:54:49.507316 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.507217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s" event={"ID":"70fc413e-37bc-45df-b14a-21df17ea52fd","Type":"ContainerDied","Data":"3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e"}
Sep 19 08:54:49.507316 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.507221    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"
Sep 19 08:54:49.507316 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.507257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s" event={"ID":"70fc413e-37bc-45df-b14a-21df17ea52fd","Type":"ContainerDied","Data":"b2b4beaa8a7e57bb84fcd20d7b961b551d2c656d3b5de05874f3cab8ace26fd9"}
Sep 19 08:54:49.507316 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.507282    2577 scope.go:117] "RemoveContainer" containerID="3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e"
Sep 19 08:54:49.517241 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.517222    2577 scope.go:117] "RemoveContainer" containerID="3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e"
Sep 19 08:54:49.517552 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:54:49.517531    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e\": container with ID starting with 3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e not found: ID does not exist" containerID="3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e"
Sep 19 08:54:49.517603 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.517559    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e"} err="failed to get container status \"3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e\": rpc error: code = NotFound desc = could not find container \"3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e\": container with ID starting with 3e51c51724d4fea85700cf9982dc97d77c00e7e2b64870609d36302eb609c65e not found: ID does not exist"
Sep 19 08:54:49.530327 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.530303    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"]
Sep 19 08:54:49.535889 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.535832    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-qwq9s"]
Sep 19 08:54:49.584208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.584185    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:54:49.584208 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.584204    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70fc413e-37bc-45df-b14a-21df17ea52fd-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:54:49.584333 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.584212    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s45s\" (UniqueName: \"kubernetes.io/projected/70fc413e-37bc-45df-b14a-21df17ea52fd-kube-api-access-9s45s\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:54:49.584333 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.584221    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70fc413e-37bc-45df-b14a-21df17ea52fd-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:54:49.744901 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:49.744859    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:54:50.504200 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:50.504167    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70fc413e-37bc-45df-b14a-21df17ea52fd" path="/var/lib/kubelet/pods/70fc413e-37bc-45df-b14a-21df17ea52fd/volumes"
Sep 19 08:54:59.744690 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:54:59.744655    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:55:04.028823 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:04.028757    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" containerName="main" containerID="cri-o://b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325" gracePeriod=45
Sep 19 08:55:09.744259 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:09.744219    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:55:09.744672 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:09.744329    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:55:19.743982 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:19.743936    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:55:29.744472 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:29.744429    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:55:34.279017 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.278992    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:55:34.359570 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.359541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9856df54-34ac-4784-99d7-beb7d90811f4-tls-certs\") pod \"9856df54-34ac-4784-99d7-beb7d90811f4\" (UID: \"9856df54-34ac-4784-99d7-beb7d90811f4\") "
Sep 19 08:55:34.359742 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.359634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w4qk\" (UniqueName: \"kubernetes.io/projected/9856df54-34ac-4784-99d7-beb7d90811f4-kube-api-access-6w4qk\") pod \"9856df54-34ac-4784-99d7-beb7d90811f4\" (UID: \"9856df54-34ac-4784-99d7-beb7d90811f4\") "
Sep 19 08:55:34.361689 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.361654    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9856df54-34ac-4784-99d7-beb7d90811f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9856df54-34ac-4784-99d7-beb7d90811f4" (UID: "9856df54-34ac-4784-99d7-beb7d90811f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:55:34.361800 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.361741    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9856df54-34ac-4784-99d7-beb7d90811f4-kube-api-access-6w4qk" (OuterVolumeSpecName: "kube-api-access-6w4qk") pod "9856df54-34ac-4784-99d7-beb7d90811f4" (UID: "9856df54-34ac-4784-99d7-beb7d90811f4"). InnerVolumeSpecName "kube-api-access-6w4qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:55:34.460720 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.460684    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w4qk\" (UniqueName: \"kubernetes.io/projected/9856df54-34ac-4784-99d7-beb7d90811f4-kube-api-access-6w4qk\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:55:34.460720 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.460713    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9856df54-34ac-4784-99d7-beb7d90811f4-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:55:34.643039 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.642967    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:55:34.651235 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.651214    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 08:55:34.746141 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.746106    2577 generic.go:356] "Generic (PLEG): container finished" podID="9856df54-34ac-4784-99d7-beb7d90811f4" containerID="b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325" exitCode=0
Sep 19 08:55:34.746330 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.746160    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"
Sep 19 08:55:34.746330 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.746166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" event={"ID":"9856df54-34ac-4784-99d7-beb7d90811f4","Type":"ContainerDied","Data":"b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325"}
Sep 19 08:55:34.746330 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.746218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph" event={"ID":"9856df54-34ac-4784-99d7-beb7d90811f4","Type":"ContainerDied","Data":"99434d1af3b5c000e4c11ce326a0564c719c87f94eb548abf1f698e9c8c1944b"}
Sep 19 08:55:34.746330 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.746246    2577 scope.go:117] "RemoveContainer" containerID="b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325"
Sep 19 08:55:34.754763 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.754744    2577 scope.go:117] "RemoveContainer" containerID="b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325"
Sep 19 08:55:34.755017 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:55:34.754998    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325\": container with ID starting with b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325 not found: ID does not exist" containerID="b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325"
Sep 19 08:55:34.755081 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.755022    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325"} err="failed to get container status \"b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325\": rpc error: code = NotFound desc = could not find container \"b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325\": container with ID starting with b345543d49405a94138edb93549dbea870c109a411e52923edd369fcc3f63325 not found: ID does not exist"
Sep 19 08:55:34.764769 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.764746    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"]
Sep 19 08:55:34.768858 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:34.768838    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schekchph"]
Sep 19 08:55:36.503827 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:36.503790    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" path="/var/lib/kubelet/pods/9856df54-34ac-4784-99d7-beb7d90811f4/volumes"
Sep 19 08:55:41.573630 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.573595    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"]
Sep 19 08:55:41.574144 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.574129    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" containerName="main"
Sep 19 08:55:41.574204 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.574149    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" containerName="main"
Sep 19 08:55:41.574204 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.574172    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52efe57b-6c93-4f91-a1ac-74b2153842a7" containerName="main"
Sep 19 08:55:41.574204 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.574181    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="52efe57b-6c93-4f91-a1ac-74b2153842a7" containerName="main"
Sep 19 08:55:41.574313 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.574213    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70fc413e-37bc-45df-b14a-21df17ea52fd" containerName="main"
Sep 19 08:55:41.574313 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.574224    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="70fc413e-37bc-45df-b14a-21df17ea52fd" containerName="main"
Sep 19 08:55:41.574313 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.574302    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="52efe57b-6c93-4f91-a1ac-74b2153842a7" containerName="main"
Sep 19 08:55:41.574313 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.574312    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="70fc413e-37bc-45df-b14a-21df17ea52fd" containerName="main"
Sep 19 08:55:41.574541 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.574322    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9856df54-34ac-4784-99d7-beb7d90811f4" containerName="main"
Sep 19 08:55:41.579484 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.579462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.583310 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.583285    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 19 08:55:41.583458 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.583286    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-xrzv2\""
Sep 19 08:55:41.583458 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.583285    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 19 08:55:41.583582 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.583294    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 19 08:55:41.587906 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.587881    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"]
Sep 19 08:55:41.618212 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.618179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.618390 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.618226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.618390 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.618248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e113c60-b538-47f4-a763-3856b3d4d520-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.618390 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.618314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmvt\" (UniqueName: \"kubernetes.io/projected/2e113c60-b538-47f4-a763-3856b3d4d520-kube-api-access-bfmvt\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.618390 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.618346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.618554 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.618396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.719700 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.719662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.719885 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.719760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.719885 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.719797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.719885 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.719823    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e113c60-b538-47f4-a763-3856b3d4d520-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.719885 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.719843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmvt\" (UniqueName: \"kubernetes.io/projected/2e113c60-b538-47f4-a763-3856b3d4d520-kube-api-access-bfmvt\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.719885 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.719866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.720201 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.720177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.720268 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.720204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.720268 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.720236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.722031 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.722011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.722228 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.722210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e113c60-b538-47f4-a763-3856b3d4d520-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.728203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.728180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmvt\" (UniqueName: \"kubernetes.io/projected/2e113c60-b538-47f4-a763-3856b3d4d520-kube-api-access-bfmvt\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:41.891630 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:41.891589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:42.175230 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:42.175193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"]
Sep 19 08:55:42.177003 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:55:42.176979    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e113c60_b538_47f4_a763_3856b3d4d520.slice/crio-f5e5c216c1f422f57a8b887df56aa7685999113979d825c329cc469d1a77a9e3 WatchSource:0}: Error finding container f5e5c216c1f422f57a8b887df56aa7685999113979d825c329cc469d1a77a9e3: Status 404 returned error can't find the container with id f5e5c216c1f422f57a8b887df56aa7685999113979d825c329cc469d1a77a9e3
Sep 19 08:55:42.791051 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:42.791012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n" event={"ID":"2e113c60-b538-47f4-a763-3856b3d4d520","Type":"ContainerStarted","Data":"dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666"}
Sep 19 08:55:42.791540 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:42.791072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n" event={"ID":"2e113c60-b538-47f4-a763-3856b3d4d520","Type":"ContainerStarted","Data":"f5e5c216c1f422f57a8b887df56aa7685999113979d825c329cc469d1a77a9e3"}
Sep 19 08:55:42.816644 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:42.816582    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n" podStartSLOduration=1.816561496 podStartE2EDuration="1.816561496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:55:42.815089406 +0000 UTC m=+2110.496672842" watchObservedRunningTime="2026-09-19 08:55:42.816561496 +0000 UTC m=+2110.498144930"
Sep 19 08:55:42.892008 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:42.891970    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:51.891971 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:51.891936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:51.904835 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:51.904805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:55:52.868091 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:55:52.868061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:56:23.965519 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:23.965424    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"]
Sep 19 08:56:23.969120 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:23.969103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:23.972070 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:23.972051    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 19 08:56:23.980593 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:23.980568    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"]
Sep 19 08:56:24.101197 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.101145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.101197 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.101202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.101474 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.101291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.101474 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.101323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.101474 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.101343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.101474 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.101401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9l7\" (UniqueName: \"kubernetes.io/projected/8b4a64eb-7b63-4544-86b4-e43abfce3aff-kube-api-access-8r9l7\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.202020 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.201985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.202203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.202029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.202203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.202058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.202203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.202084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.202203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.202110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.202203 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.202144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9l7\" (UniqueName: \"kubernetes.io/projected/8b4a64eb-7b63-4544-86b4-e43abfce3aff-kube-api-access-8r9l7\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.202557 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.202526    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.202690 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.202612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.202877 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.202852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.204484 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.204452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.204746 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.204725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.216329 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.216264    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9l7\" (UniqueName: \"kubernetes.io/projected/8b4a64eb-7b63-4544-86b4-e43abfce3aff-kube-api-access-8r9l7\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.280885 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.280849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:24.557207 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:24.557176    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"]
Sep 19 08:56:24.558808 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:56:24.558777    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4a64eb_7b63_4544_86b4_e43abfce3aff.slice/crio-77c4245f02e98d65c3575c00baccf72ded40420ef2068a341792405d99e1123e WatchSource:0}: Error finding container 77c4245f02e98d65c3575c00baccf72ded40420ef2068a341792405d99e1123e: Status 404 returned error can't find the container with id 77c4245f02e98d65c3575c00baccf72ded40420ef2068a341792405d99e1123e
Sep 19 08:56:25.027438 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:25.027399    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg" event={"ID":"8b4a64eb-7b63-4544-86b4-e43abfce3aff","Type":"ContainerStarted","Data":"f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9"}
Sep 19 08:56:25.027933 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:25.027470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg" event={"ID":"8b4a64eb-7b63-4544-86b4-e43abfce3aff","Type":"ContainerStarted","Data":"77c4245f02e98d65c3575c00baccf72ded40420ef2068a341792405d99e1123e"}
Sep 19 08:56:25.050476 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:25.050429    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg" podStartSLOduration=2.050414148 podStartE2EDuration="2.050414148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:56:25.048830561 +0000 UTC m=+2152.730413994" watchObservedRunningTime="2026-09-19 08:56:25.050414148 +0000 UTC m=+2152.731997580"
Sep 19 08:56:25.281961 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:25.281881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:34.281578 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:34.281531    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:34.294923 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:34.294895    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:35.097713 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:35.097683    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:56:59.608507 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:59.608470    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"]
Sep 19 08:56:59.618691 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:56:59.618655    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:56:59.618691 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:56:59.618691 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:56:59.618691 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9" cmd=["/bin/sleep","15"]
Sep 19 08:56:59.618994 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:59.618707    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg" message="" err=<
Sep 19 08:56:59.618994 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:56:59.618994 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:56:59.618994 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:56:59.618994 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:56:59.618719    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:56:59.618994 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:56:59.618994 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:56:59.618994 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg" podUID="8b4a64eb-7b63-4544-86b4-e43abfce3aff" containerName="main" containerID="cri-o://f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9"
Sep 19 08:56:59.618994 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:59.618763    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg" podUID="8b4a64eb-7b63-4544-86b4-e43abfce3aff" containerName="main" containerID="cri-o://f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9" gracePeriod=60
Sep 19 08:56:59.969458 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:56:59.969433    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:57:00.122175 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-home\") pod \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") "
Sep 19 08:57:00.122350 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122229    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-dshm\") pod \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") "
Sep 19 08:57:00.122350 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122267    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tls-certs\") pod \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") "
Sep 19 08:57:00.122350 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122305    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-model-cache\") pod \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") "
Sep 19 08:57:00.122350 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tmp-dir\") pod \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") "
Sep 19 08:57:00.122608 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r9l7\" (UniqueName: \"kubernetes.io/projected/8b4a64eb-7b63-4544-86b4-e43abfce3aff-kube-api-access-8r9l7\") pod \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\" (UID: \"8b4a64eb-7b63-4544-86b4-e43abfce3aff\") "
Sep 19 08:57:00.122608 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122466    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-home" (OuterVolumeSpecName: "home") pod "8b4a64eb-7b63-4544-86b4-e43abfce3aff" (UID: "8b4a64eb-7b63-4544-86b4-e43abfce3aff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:00.122719 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b4a64eb-7b63-4544-86b4-e43abfce3aff" (UID: "8b4a64eb-7b63-4544-86b4-e43abfce3aff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:00.122719 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122647    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b4a64eb-7b63-4544-86b4-e43abfce3aff" (UID: "8b4a64eb-7b63-4544-86b4-e43abfce3aff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:00.122831 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122747    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:00.122831 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122760    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:00.122831 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.122798    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:00.124751 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.124725    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b4a64eb-7b63-4544-86b4-e43abfce3aff" (UID: "8b4a64eb-7b63-4544-86b4-e43abfce3aff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:57:00.124888 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.124870    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-dshm" (OuterVolumeSpecName: "dshm") pod "8b4a64eb-7b63-4544-86b4-e43abfce3aff" (UID: "8b4a64eb-7b63-4544-86b4-e43abfce3aff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:00.124948 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.124898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4a64eb-7b63-4544-86b4-e43abfce3aff-kube-api-access-8r9l7" (OuterVolumeSpecName: "kube-api-access-8r9l7") pod "8b4a64eb-7b63-4544-86b4-e43abfce3aff" (UID: "8b4a64eb-7b63-4544-86b4-e43abfce3aff"). InnerVolumeSpecName "kube-api-access-8r9l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:57:00.223394 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.223291    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r9l7\" (UniqueName: \"kubernetes.io/projected/8b4a64eb-7b63-4544-86b4-e43abfce3aff-kube-api-access-8r9l7\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:00.223394 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.223324    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b4a64eb-7b63-4544-86b4-e43abfce3aff-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:00.223394 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.223338    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b4a64eb-7b63-4544-86b4-e43abfce3aff-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:00.224341 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.224313    2577 generic.go:356] "Generic (PLEG): container finished" podID="8b4a64eb-7b63-4544-86b4-e43abfce3aff" containerID="f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9" exitCode=0
Sep 19 08:57:00.224490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.224382    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"
Sep 19 08:57:00.224490 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.224405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg" event={"ID":"8b4a64eb-7b63-4544-86b4-e43abfce3aff","Type":"ContainerDied","Data":"f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9"}
Sep 19 08:57:00.224573 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.224489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg" event={"ID":"8b4a64eb-7b63-4544-86b4-e43abfce3aff","Type":"ContainerDied","Data":"77c4245f02e98d65c3575c00baccf72ded40420ef2068a341792405d99e1123e"}
Sep 19 08:57:00.224573 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.224512    2577 scope.go:117] "RemoveContainer" containerID="f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9"
Sep 19 08:57:00.234483 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.234464    2577 scope.go:117] "RemoveContainer" containerID="f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9"
Sep 19 08:57:00.234748 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:57:00.234730    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9\": container with ID starting with f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9 not found: ID does not exist" containerID="f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9"
Sep 19 08:57:00.234801 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.234757    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9"} err="failed to get container status \"f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9\": rpc error: code = NotFound desc = could not find container \"f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9\": container with ID starting with f08a7b2349c8a3d2f4b5ced78d145f9cf2784925074e5154d0eda9165e538cd9 not found: ID does not exist"
Sep 19 08:57:00.247306 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.247280    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"]
Sep 19 08:57:00.253866 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.253843    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-x7jdg"]
Sep 19 08:57:00.504197 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:00.504120    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b4a64eb-7b63-4544-86b4-e43abfce3aff" path="/var/lib/kubelet/pods/8b4a64eb-7b63-4544-86b4-e43abfce3aff/volumes"
Sep 19 08:57:48.493464 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.493072    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"]
Sep 19 08:57:48.504892 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:57:48.504859    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:57:48.504892 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:57:48.504892 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:57:48.504892 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666" cmd=["/bin/sleep","15"]
Sep 19 08:57:48.505179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.504910    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n" message="" err=<
Sep 19 08:57:48.505179 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:57:48.505179 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:57:48.505179 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 08:57:48.505179 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:57:48.504922    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:57:48.505179 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:57:48.505179 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 08:57:48.505179 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n" podUID="2e113c60-b538-47f4-a763-3856b3d4d520" containerName="main" containerID="cri-o://dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666"
Sep 19 08:57:48.505179 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.504970    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n" podUID="2e113c60-b538-47f4-a763-3856b3d4d520" containerName="main" containerID="cri-o://dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666" gracePeriod=60
Sep 19 08:57:48.858960 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.858935    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:57:48.896834 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.896805    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfmvt\" (UniqueName: \"kubernetes.io/projected/2e113c60-b538-47f4-a763-3856b3d4d520-kube-api-access-bfmvt\") pod \"2e113c60-b538-47f4-a763-3856b3d4d520\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") "
Sep 19 08:57:48.897029 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.896864    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-home\") pod \"2e113c60-b538-47f4-a763-3856b3d4d520\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") "
Sep 19 08:57:48.897029 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.896892    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e113c60-b538-47f4-a763-3856b3d4d520-tls-certs\") pod \"2e113c60-b538-47f4-a763-3856b3d4d520\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") "
Sep 19 08:57:48.897029 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.896907    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-model-cache\") pod \"2e113c60-b538-47f4-a763-3856b3d4d520\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") "
Sep 19 08:57:48.897029 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.896977    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-dshm\") pod \"2e113c60-b538-47f4-a763-3856b3d4d520\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") "
Sep 19 08:57:48.897029 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.897022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-tmp-dir\") pod \"2e113c60-b538-47f4-a763-3856b3d4d520\" (UID: \"2e113c60-b538-47f4-a763-3856b3d4d520\") "
Sep 19 08:57:48.897320 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.897136    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-home" (OuterVolumeSpecName: "home") pod "2e113c60-b538-47f4-a763-3856b3d4d520" (UID: "2e113c60-b538-47f4-a763-3856b3d4d520"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:48.897320 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.897276    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:48.897458 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.897390    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e113c60-b538-47f4-a763-3856b3d4d520" (UID: "2e113c60-b538-47f4-a763-3856b3d4d520"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:48.897520 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.897501    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e113c60-b538-47f4-a763-3856b3d4d520" (UID: "2e113c60-b538-47f4-a763-3856b3d4d520"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:48.899220 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.899192    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e113c60-b538-47f4-a763-3856b3d4d520-kube-api-access-bfmvt" (OuterVolumeSpecName: "kube-api-access-bfmvt") pod "2e113c60-b538-47f4-a763-3856b3d4d520" (UID: "2e113c60-b538-47f4-a763-3856b3d4d520"). InnerVolumeSpecName "kube-api-access-bfmvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:57:48.899333 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.899215    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-dshm" (OuterVolumeSpecName: "dshm") pod "2e113c60-b538-47f4-a763-3856b3d4d520" (UID: "2e113c60-b538-47f4-a763-3856b3d4d520"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:57:48.899536 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.899517    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e113c60-b538-47f4-a763-3856b3d4d520-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e113c60-b538-47f4-a763-3856b3d4d520" (UID: "2e113c60-b538-47f4-a763-3856b3d4d520"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:57:48.997671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.997635    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:48.997671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.997663    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:48.997671 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.997672    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfmvt\" (UniqueName: \"kubernetes.io/projected/2e113c60-b538-47f4-a763-3856b3d4d520-kube-api-access-bfmvt\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:48.997894 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.997681    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e113c60-b538-47f4-a763-3856b3d4d520-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:48.997894 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:48.997691    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e113c60-b538-47f4-a763-3856b3d4d520-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 08:57:49.480433 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:49.480392    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e113c60-b538-47f4-a763-3856b3d4d520" containerID="dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666" exitCode=0
Sep 19 08:57:49.480638 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:49.480516    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"
Sep 19 08:57:49.480638 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:49.480552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n" event={"ID":"2e113c60-b538-47f4-a763-3856b3d4d520","Type":"ContainerDied","Data":"dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666"}
Sep 19 08:57:49.480638 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:49.480630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n" event={"ID":"2e113c60-b538-47f4-a763-3856b3d4d520","Type":"ContainerDied","Data":"f5e5c216c1f422f57a8b887df56aa7685999113979d825c329cc469d1a77a9e3"}
Sep 19 08:57:49.480796 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:49.480661    2577 scope.go:117] "RemoveContainer" containerID="dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666"
Sep 19 08:57:49.492209 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:49.492191    2577 scope.go:117] "RemoveContainer" containerID="dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666"
Sep 19 08:57:49.492538 ip-10-0-131-33 kubenswrapper[2577]: E0919 08:57:49.492521    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666\": container with ID starting with dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666 not found: ID does not exist" containerID="dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666"
Sep 19 08:57:49.492615 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:49.492548    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666"} err="failed to get container status \"dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666\": rpc error: code = NotFound desc = could not find container \"dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666\": container with ID starting with dc46c6e97cc408534252989fbecc686c6faa3977bd1710b0275ac308a83c6666 not found: ID does not exist"
Sep 19 08:57:49.505492 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:49.505468    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"]
Sep 19 08:57:49.510350 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:49.510329    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969c7m8n"]
Sep 19 08:57:50.503415 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:57:50.503358    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e113c60-b538-47f4-a763-3856b3d4d520" path="/var/lib/kubelet/pods/2e113c60-b538-47f4-a763-3856b3d4d520/volumes"
Sep 19 08:58:47.689214 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.689177    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"]
Sep 19 08:58:47.689841 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.689730    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e113c60-b538-47f4-a763-3856b3d4d520" containerName="main"
Sep 19 08:58:47.689841 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.689750    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e113c60-b538-47f4-a763-3856b3d4d520" containerName="main"
Sep 19 08:58:47.689841 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.689782    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b4a64eb-7b63-4544-86b4-e43abfce3aff" containerName="main"
Sep 19 08:58:47.689841 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.689790    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b4a64eb-7b63-4544-86b4-e43abfce3aff" containerName="main"
Sep 19 08:58:47.690063 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.689881    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e113c60-b538-47f4-a763-3856b3d4d520" containerName="main"
Sep 19 08:58:47.690063 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.689897    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b4a64eb-7b63-4544-86b4-e43abfce3aff" containerName="main"
Sep 19 08:58:47.694239 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.694216    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.696788 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.696760    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 19 08:58:47.696925 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.696806    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 19 08:58:47.697092 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.697076    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 19 08:58:47.697172 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.697129    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:58:47.697243 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.697223    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-hwhm9\""
Sep 19 08:58:47.697930 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.697908    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 19 08:58:47.705396 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.705355    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"]
Sep 19 08:58:47.787894 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.787860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.787894 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.787893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.788129 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.787918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.788129 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.787960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.788129 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.787999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.788129 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.788035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5qf\" (UniqueName: \"kubernetes.io/projected/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kube-api-access-dk5qf\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.788129 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.788076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.788129 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.788107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.888997 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.888958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889219 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889219 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889219 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889219 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889219 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889524 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889524 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5qf\" (UniqueName: \"kubernetes.io/projected/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kube-api-access-dk5qf\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889524 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889685 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889743 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889809 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.889904 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.889886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.892105 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.892080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.892230 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.892167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:47.901414 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:47.901356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5qf\" (UniqueName: \"kubernetes.io/projected/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kube-api-access-dk5qf\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:48.006265 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:48.006172    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:48.283165 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:48.283137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"]
Sep 19 08:58:48.284856 ip-10-0-131-33 kubenswrapper[2577]: W0919 08:58:48.284823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32e7c2a_2f7a_4fd1_90f4_9e9d712e2206.slice/crio-aec25e7201a4a25911dec63d16e4e42aa5df5b6035509311c55452e98d7dfa7d WatchSource:0}: Error finding container aec25e7201a4a25911dec63d16e4e42aa5df5b6035509311c55452e98d7dfa7d: Status 404 returned error can't find the container with id aec25e7201a4a25911dec63d16e4e42aa5df5b6035509311c55452e98d7dfa7d
Sep 19 08:58:48.790462 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:48.790419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" event={"ID":"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206","Type":"ContainerStarted","Data":"d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895"}
Sep 19 08:58:48.790917 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:48.790493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" event={"ID":"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206","Type":"ContainerStarted","Data":"aec25e7201a4a25911dec63d16e4e42aa5df5b6035509311c55452e98d7dfa7d"}
Sep 19 08:58:51.809458 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:51.809420    2577 generic.go:356] "Generic (PLEG): container finished" podID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerID="d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895" exitCode=0
Sep 19 08:58:51.809872 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:51.809494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" event={"ID":"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206","Type":"ContainerDied","Data":"d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895"}
Sep 19 08:58:52.817338 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:52.817302    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" event={"ID":"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206","Type":"ContainerStarted","Data":"a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d"}
Sep 19 08:58:52.843036 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:52.842971    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" podStartSLOduration=3.020313062 podStartE2EDuration="5.842950432s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.82263737s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:58:52.839432686 +0000 UTC m=+2300.521016120" watchObservedRunningTime="2026-09-19 08:58:52.842950432 +0000 UTC m=+2300.524533867"
Sep 19 08:58:53.007016 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:53.006972    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:58.007257 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:58.007208    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 08:58:58.009049 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:58:58.009008    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 19 08:59:08.007584 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:59:08.007541    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 19 08:59:18.007041 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:59:18.006936    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 19 08:59:28.007059 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:59:28.007006    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 19 08:59:38.006701 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:59:38.006647    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 19 08:59:48.006805 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:59:48.006761    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 19 08:59:58.006994 ip-10-0-131-33 kubenswrapper[2577]: I0919 08:59:58.006948    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Sep 19 09:00:08.017866 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:08.017832    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 09:00:08.026573 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:08.026542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 09:00:34.681697 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:34.681668    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:00:34.689917 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:34.689896    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:00:34.940527 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:34.940446    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"]
Sep 19 09:00:49.964040 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:49.963935    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main" containerID="cri-o://a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d" gracePeriod=45
Sep 19 09:00:50.624723 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.624693    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 09:00:50.663128 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663063    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tmp-dir\") pod \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") "
Sep 19 09:00:50.663128 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663092    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-home\") pod \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") "
Sep 19 09:00:50.663128 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663120    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-cabundle-cert\") pod \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") "
Sep 19 09:00:50.663418 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kserve-provision-location\") pod \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") "
Sep 19 09:00:50.663418 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tls-certs\") pod \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") "
Sep 19 09:00:50.663418 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk5qf\" (UniqueName: \"kubernetes.io/projected/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kube-api-access-dk5qf\") pod \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") "
Sep 19 09:00:50.663587 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663420    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-model-cache\") pod \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") "
Sep 19 09:00:50.663587 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663478    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-dshm\") pod \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\" (UID: \"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206\") "
Sep 19 09:00:50.663587 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663539    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-home" (OuterVolumeSpecName: "home") pod "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" (UID: "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:00:50.663753 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663599    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" (UID: "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:00:50.663812 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663796    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:00:50.663868 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663815    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:00:50.663868 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.663821    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-model-cache" (OuterVolumeSpecName: "model-cache") pod "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" (UID: "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:00:50.665797 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.665764    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-dshm" (OuterVolumeSpecName: "dshm") pod "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" (UID: "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:00:50.665913 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.665888    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" (UID: "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:00:50.665965 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.665898    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kube-api-access-dk5qf" (OuterVolumeSpecName: "kube-api-access-dk5qf") pod "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" (UID: "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206"). InnerVolumeSpecName "kube-api-access-dk5qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:00:50.680853 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.680825    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" (UID: "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:00:50.688594 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.688570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" (UID: "c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:00:50.764492 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.764466    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:00:50.764492 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.764489    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:00:50.764631 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.764501    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:00:50.764631 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.764510    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk5qf\" (UniqueName: \"kubernetes.io/projected/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-kube-api-access-dk5qf\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:00:50.764631 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.764518    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:00:50.764631 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:50.764526    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:00:51.481712 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.481668    2577 generic.go:356] "Generic (PLEG): container finished" podID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerID="a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d" exitCode=0
Sep 19 09:00:51.481712 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.481715    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"
Sep 19 09:00:51.482202 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.481730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" event={"ID":"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206","Type":"ContainerDied","Data":"a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d"}
Sep 19 09:00:51.482202 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.481780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl" event={"ID":"c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206","Type":"ContainerDied","Data":"aec25e7201a4a25911dec63d16e4e42aa5df5b6035509311c55452e98d7dfa7d"}
Sep 19 09:00:51.482202 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.481807    2577 scope.go:117] "RemoveContainer" containerID="a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d"
Sep 19 09:00:51.491272 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.491248    2577 scope.go:117] "RemoveContainer" containerID="d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895"
Sep 19 09:00:51.499555 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.499536    2577 scope.go:117] "RemoveContainer" containerID="a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d"
Sep 19 09:00:51.499826 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:00:51.499807    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d\": container with ID starting with a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d not found: ID does not exist" containerID="a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d"
Sep 19 09:00:51.499903 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.499839    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d"} err="failed to get container status \"a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d\": rpc error: code = NotFound desc = could not find container \"a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d\": container with ID starting with a2492743ab6a8d41c0e457dbcc0146c5b227a6055e4f19df45c03c96b2e6728d not found: ID does not exist"
Sep 19 09:00:51.499903 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.499862    2577 scope.go:117] "RemoveContainer" containerID="d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895"
Sep 19 09:00:51.500131 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:00:51.500110    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895\": container with ID starting with d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895 not found: ID does not exist" containerID="d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895"
Sep 19 09:00:51.500172 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.500138    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895"} err="failed to get container status \"d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895\": rpc error: code = NotFound desc = could not find container \"d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895\": container with ID starting with d873ebd8ab919c905dd2bcdf382efd726be81af800278f7c0eec1814e3e42895 not found: ID does not exist"
Sep 19 09:00:51.505140 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.505117    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"]
Sep 19 09:00:51.510991 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:51.510965    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-gbhfl"]
Sep 19 09:00:52.503055 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:00:52.503024    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" path="/var/lib/kubelet/pods/c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206/volumes"
Sep 19 09:01:37.393237 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.393202    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"]
Sep 19 09:01:37.393787 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.393767    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="storage-initializer"
Sep 19 09:01:37.393869 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.393790    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="storage-initializer"
Sep 19 09:01:37.393869 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.393842    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main"
Sep 19 09:01:37.393869 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.393851    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main"
Sep 19 09:01:37.394041 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.393962    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32e7c2a-2f7a-4fd1-90f4-9e9d712e2206" containerName="main"
Sep 19 09:01:37.398420 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.398398    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.401375 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.401335    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 19 09:01:37.401819 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.401791    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 19 09:01:37.402068 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.402049    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-mm7r7\""
Sep 19 09:01:37.402701 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.402678    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 19 09:01:37.402971 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.402948    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:01:37.403223 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.403202    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 19 09:01:37.413628 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.413602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"]
Sep 19 09:01:37.434286 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.434256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14e0704-2f62-42aa-9243-f16cd486b5ae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.434286 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.434288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14e0704-2f62-42aa-9243-f16cd486b5ae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.434521 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.434308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.434521 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.434327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.434521 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.434385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.434521 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.434456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.434521 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.434501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbtvw\" (UniqueName: \"kubernetes.io/projected/f14e0704-2f62-42aa-9243-f16cd486b5ae-kube-api-access-vbtvw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.434708 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.434554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.535491 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14e0704-2f62-42aa-9243-f16cd486b5ae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.535491 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14e0704-2f62-42aa-9243-f16cd486b5ae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.535764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.535764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.535764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.535764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.535764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbtvw\" (UniqueName: \"kubernetes.io/projected/f14e0704-2f62-42aa-9243-f16cd486b5ae-kube-api-access-vbtvw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.535764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.536087 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.536087 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.535980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.536199 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.536082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.536298 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.536279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.536423 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.536401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14e0704-2f62-42aa-9243-f16cd486b5ae-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.538579 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.538556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.538726 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.538710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14e0704-2f62-42aa-9243-f16cd486b5ae-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.544133 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.544104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbtvw\" (UniqueName: \"kubernetes.io/projected/f14e0704-2f62-42aa-9243-f16cd486b5ae-kube-api-access-vbtvw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.713111 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.713027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:37.992942 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:37.992857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"]
Sep 19 09:01:37.996132 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:01:37.996104    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14e0704_2f62_42aa_9243_f16cd486b5ae.slice/crio-198f9beeae732446f75211be33f46bc16073110b18ca7166e683275530fde3cd WatchSource:0}: Error finding container 198f9beeae732446f75211be33f46bc16073110b18ca7166e683275530fde3cd: Status 404 returned error can't find the container with id 198f9beeae732446f75211be33f46bc16073110b18ca7166e683275530fde3cd
Sep 19 09:01:38.735349 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:38.735309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" event={"ID":"f14e0704-2f62-42aa-9243-f16cd486b5ae","Type":"ContainerStarted","Data":"7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d"}
Sep 19 09:01:38.735786 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:38.735395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" event={"ID":"f14e0704-2f62-42aa-9243-f16cd486b5ae","Type":"ContainerStarted","Data":"198f9beeae732446f75211be33f46bc16073110b18ca7166e683275530fde3cd"}
Sep 19 09:01:41.758722 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:41.758690    2577 generic.go:356] "Generic (PLEG): container finished" podID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerID="7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d" exitCode=0
Sep 19 09:01:41.759239 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:41.758770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" event={"ID":"f14e0704-2f62-42aa-9243-f16cd486b5ae","Type":"ContainerDied","Data":"7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d"}
Sep 19 09:01:42.773395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:42.773339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" event={"ID":"f14e0704-2f62-42aa-9243-f16cd486b5ae","Type":"ContainerStarted","Data":"81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060"}
Sep 19 09:01:42.797942 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:42.797471    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podStartSLOduration=3.105653636 podStartE2EDuration="5.797451816s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.69179818s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:01:42.796236122 +0000 UTC m=+2470.477819555" watchObservedRunningTime="2026-09-19 09:01:42.797451816 +0000 UTC m=+2470.479035258"
Sep 19 09:01:43.713168 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:43.713121    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:47.713231 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:47.713189    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:01:47.715013 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:47.714983    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 19 09:01:57.714228 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:01:57.714176    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 19 09:02:07.713695 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:02:07.713644    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 19 09:02:17.714107 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:02:17.714004    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 19 09:02:27.714232 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:02:27.714169    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 19 09:02:37.713799 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:02:37.713751    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 19 09:02:47.713971 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:02:47.713925    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 19 09:02:57.723655 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:02:57.723617    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:02:57.732039 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:02:57.732012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:03:22.388541 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:22.388498    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"]
Sep 19 09:03:37.419143 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:37.419067    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main" containerID="cri-o://81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060" gracePeriod=45
Sep 19 09:03:37.725004 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:37.724906    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Sep 19 09:03:37.984081 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:37.984010    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:03:38.099247 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099217    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-home\") pod \"f14e0704-2f62-42aa-9243-f16cd486b5ae\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") "
Sep 19 09:03:38.099472 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099262    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14e0704-2f62-42aa-9243-f16cd486b5ae-tls-certs\") pod \"f14e0704-2f62-42aa-9243-f16cd486b5ae\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") "
Sep 19 09:03:38.099472 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbtvw\" (UniqueName: \"kubernetes.io/projected/f14e0704-2f62-42aa-9243-f16cd486b5ae-kube-api-access-vbtvw\") pod \"f14e0704-2f62-42aa-9243-f16cd486b5ae\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") "
Sep 19 09:03:38.099472 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099324    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-dshm\") pod \"f14e0704-2f62-42aa-9243-f16cd486b5ae\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") "
Sep 19 09:03:38.099472 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-tmp-dir\") pod \"f14e0704-2f62-42aa-9243-f16cd486b5ae\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") "
Sep 19 09:03:38.099472 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14e0704-2f62-42aa-9243-f16cd486b5ae-cabundle-cert\") pod \"f14e0704-2f62-42aa-9243-f16cd486b5ae\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") "
Sep 19 09:03:38.099472 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-model-cache\") pod \"f14e0704-2f62-42aa-9243-f16cd486b5ae\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") "
Sep 19 09:03:38.099472 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-kserve-provision-location\") pod \"f14e0704-2f62-42aa-9243-f16cd486b5ae\" (UID: \"f14e0704-2f62-42aa-9243-f16cd486b5ae\") "
Sep 19 09:03:38.099889 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099848    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-home" (OuterVolumeSpecName: "home") pod "f14e0704-2f62-42aa-9243-f16cd486b5ae" (UID: "f14e0704-2f62-42aa-9243-f16cd486b5ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:03:38.099948 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "f14e0704-2f62-42aa-9243-f16cd486b5ae" (UID: "f14e0704-2f62-42aa-9243-f16cd486b5ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:03:38.099989 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.099960    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f14e0704-2f62-42aa-9243-f16cd486b5ae" (UID: "f14e0704-2f62-42aa-9243-f16cd486b5ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:03:38.100127 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.100085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14e0704-2f62-42aa-9243-f16cd486b5ae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f14e0704-2f62-42aa-9243-f16cd486b5ae" (UID: "f14e0704-2f62-42aa-9243-f16cd486b5ae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:03:38.101755 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.101729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14e0704-2f62-42aa-9243-f16cd486b5ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f14e0704-2f62-42aa-9243-f16cd486b5ae" (UID: "f14e0704-2f62-42aa-9243-f16cd486b5ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:03:38.101840 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.101749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-dshm" (OuterVolumeSpecName: "dshm") pod "f14e0704-2f62-42aa-9243-f16cd486b5ae" (UID: "f14e0704-2f62-42aa-9243-f16cd486b5ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:03:38.101913 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.101886    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14e0704-2f62-42aa-9243-f16cd486b5ae-kube-api-access-vbtvw" (OuterVolumeSpecName: "kube-api-access-vbtvw") pod "f14e0704-2f62-42aa-9243-f16cd486b5ae" (UID: "f14e0704-2f62-42aa-9243-f16cd486b5ae"). InnerVolumeSpecName "kube-api-access-vbtvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:03:38.124330 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.124302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f14e0704-2f62-42aa-9243-f16cd486b5ae" (UID: "f14e0704-2f62-42aa-9243-f16cd486b5ae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:03:38.200344 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.200310    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:03:38.200344 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.200339    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f14e0704-2f62-42aa-9243-f16cd486b5ae-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:03:38.200344 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.200351    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbtvw\" (UniqueName: \"kubernetes.io/projected/f14e0704-2f62-42aa-9243-f16cd486b5ae-kube-api-access-vbtvw\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:03:38.200597 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.200382    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:03:38.200597 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.200392    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:03:38.200597 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.200400    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f14e0704-2f62-42aa-9243-f16cd486b5ae-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:03:38.200597 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.200408    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:03:38.200597 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.200416    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f14e0704-2f62-42aa-9243-f16cd486b5ae-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:03:38.431929 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.431892    2577 generic.go:356] "Generic (PLEG): container finished" podID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerID="81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060" exitCode=0
Sep 19 09:03:38.432395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.431942    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"
Sep 19 09:03:38.432395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.431947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" event={"ID":"f14e0704-2f62-42aa-9243-f16cd486b5ae","Type":"ContainerDied","Data":"81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060"}
Sep 19 09:03:38.432395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.432006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88" event={"ID":"f14e0704-2f62-42aa-9243-f16cd486b5ae","Type":"ContainerDied","Data":"198f9beeae732446f75211be33f46bc16073110b18ca7166e683275530fde3cd"}
Sep 19 09:03:38.432395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.432025    2577 scope.go:117] "RemoveContainer" containerID="81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060"
Sep 19 09:03:38.441131 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.441113    2577 scope.go:117] "RemoveContainer" containerID="7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d"
Sep 19 09:03:38.449235 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.449217    2577 scope.go:117] "RemoveContainer" containerID="81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060"
Sep 19 09:03:38.449522 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:03:38.449506    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060\": container with ID starting with 81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060 not found: ID does not exist" containerID="81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060"
Sep 19 09:03:38.449571 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.449531    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060"} err="failed to get container status \"81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060\": rpc error: code = NotFound desc = could not find container \"81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060\": container with ID starting with 81e0a21783a7fd253855637e234b4bfaede30ba5af1abd395046c395b32a6060 not found: ID does not exist"
Sep 19 09:03:38.449571 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.449548    2577 scope.go:117] "RemoveContainer" containerID="7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d"
Sep 19 09:03:38.449798 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:03:38.449779    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d\": container with ID starting with 7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d not found: ID does not exist" containerID="7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d"
Sep 19 09:03:38.449835 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.449806    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d"} err="failed to get container status \"7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d\": rpc error: code = NotFound desc = could not find container \"7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d\": container with ID starting with 7a55360090b4aad9ae449073c895e7a24d4d28a04ef6eb7b6c2ff9a357a0007d not found: ID does not exist"
Sep 19 09:03:38.455447 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.455412    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"]
Sep 19 09:03:38.458529 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.458507    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-qqp88"]
Sep 19 09:03:38.503600 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:03:38.503568    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" path="/var/lib/kubelet/pods/f14e0704-2f62-42aa-9243-f16cd486b5ae/volumes"
Sep 19 09:04:21.085329 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.085244    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"]
Sep 19 09:04:21.085837 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.085651    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="storage-initializer"
Sep 19 09:04:21.085837 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.085664    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="storage-initializer"
Sep 19 09:04:21.085837 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.085684    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main"
Sep 19 09:04:21.085837 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.085689    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main"
Sep 19 09:04:21.085837 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.085748    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f14e0704-2f62-42aa-9243-f16cd486b5ae" containerName="main"
Sep 19 09:04:21.088738 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.088717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.091315 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.091290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 19 09:04:21.091951 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.091929    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 19 09:04:21.092056 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.091976    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 19 09:04:21.092056 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.092015    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:04:21.092056 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.092015    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-85zwp\""
Sep 19 09:04:21.092239 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.092018    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 19 09:04:21.101104 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.101079    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"]
Sep 19 09:04:21.158682 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.158646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.158868 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.158690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.158868 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.158727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.158868 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.158785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.158868 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.158811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2qcg\" (UniqueName: \"kubernetes.io/projected/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kube-api-access-v2qcg\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.158868 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.158835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.159096 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.158893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-home\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.159096 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.158957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-dshm\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259315 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2qcg\" (UniqueName: \"kubernetes.io/projected/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kube-api-access-v2qcg\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259315 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259600 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-home\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259600 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-dshm\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259600 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259600 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259600 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259600 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259911 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.259911 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.259876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-home\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.260039 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.260014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.260140 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.260115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.260421 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.260400    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.261870 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.261851    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-dshm\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.262325 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.262305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.267275 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.267254    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2qcg\" (UniqueName: \"kubernetes.io/projected/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kube-api-access-v2qcg\") pod \"stop-feature-test-kserve-6bfc568595-g2pqs\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.400949 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.400909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:21.688687 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:21.688656    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"]
Sep 19 09:04:21.690683 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:04:21.690651    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9e7cab_83d1_4fbe_b363_8a11db04e2bc.slice/crio-bde42b0861c5b1bc30935acd5b31ea7d167c51f5d138bbbd05663b73c76dc7ff WatchSource:0}: Error finding container bde42b0861c5b1bc30935acd5b31ea7d167c51f5d138bbbd05663b73c76dc7ff: Status 404 returned error can't find the container with id bde42b0861c5b1bc30935acd5b31ea7d167c51f5d138bbbd05663b73c76dc7ff
Sep 19 09:04:22.671520 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:22.671484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" event={"ID":"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc","Type":"ContainerStarted","Data":"6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672"}
Sep 19 09:04:22.671936 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:22.671536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" event={"ID":"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc","Type":"ContainerStarted","Data":"bde42b0861c5b1bc30935acd5b31ea7d167c51f5d138bbbd05663b73c76dc7ff"}
Sep 19 09:04:24.683603 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:24.683568    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerID="6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672" exitCode=0
Sep 19 09:04:24.684020 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:24.683626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" event={"ID":"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc","Type":"ContainerDied","Data":"6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672"}
Sep 19 09:04:25.691147 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:25.691111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" event={"ID":"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc","Type":"ContainerStarted","Data":"7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb"}
Sep 19 09:04:25.717473 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:25.717419    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podStartSLOduration=2.420476473 podStartE2EDuration="4.717402452s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.296925979s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:04:25.713658457 +0000 UTC m=+2633.395241912" watchObservedRunningTime="2026-09-19 09:04:25.717402452 +0000 UTC m=+2633.398985884"
Sep 19 09:04:26.401221 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:26.401184    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:31.401764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:31.401726    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:04:31.403399 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:31.403358    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:04:41.401938 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:41.401892    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:04:51.402067 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:04:51.402021    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:05:01.402156 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:05:01.402114    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:05:11.401706 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:05:11.401657    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:05:21.402432 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:05:21.402330    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:05:31.402234 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:05:31.402178    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:05:34.723564 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:05:34.723532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:05:34.729819 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:05:34.729792    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:05:41.402085 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:05:41.402038    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:05:51.401629 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:05:51.401583    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:06:01.401578 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:01.401531    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:06:11.401803 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:11.401761    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 19 09:06:21.411544 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:21.411508    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:06:21.420025 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:21.419986    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:06:23.156066 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:23.156028    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"]
Sep 19 09:06:38.387690 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:38.387632    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main" containerID="cri-o://7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb" gracePeriod=45
Sep 19 09:06:39.141046 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.141017    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:06:39.316341 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316231    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2qcg\" (UniqueName: \"kubernetes.io/projected/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kube-api-access-v2qcg\") pod \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") "
Sep 19 09:06:39.316341 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tmp-dir\") pod \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") "
Sep 19 09:06:39.316341 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316300    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-cabundle-cert\") pod \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") "
Sep 19 09:06:39.316341 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-dshm\") pod \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") "
Sep 19 09:06:39.316727 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316396    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tls-certs\") pod \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") "
Sep 19 09:06:39.316727 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-model-cache\") pod \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") "
Sep 19 09:06:39.316727 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316494    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kserve-provision-location\") pod \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") "
Sep 19 09:06:39.316727 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316526    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-home\") pod \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\" (UID: \"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc\") "
Sep 19 09:06:39.316727 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316707    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" (UID: "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:06:39.316985 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" (UID: "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:06:39.316985 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316856    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:06:39.316985 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.316874    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:06:39.317436 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.317407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-home" (OuterVolumeSpecName: "home") pod "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" (UID: "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:06:39.318832 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.318799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" (UID: "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:06:39.318931 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.318834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kube-api-access-v2qcg" (OuterVolumeSpecName: "kube-api-access-v2qcg") pod "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" (UID: "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc"). InnerVolumeSpecName "kube-api-access-v2qcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:06:39.319004 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.318985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-dshm" (OuterVolumeSpecName: "dshm") pod "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" (UID: "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:06:39.334419 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.334349    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" (UID: "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:06:39.340941 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.340914    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" (UID: "ee9e7cab-83d1-4fbe-b363-8a11db04e2bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:06:39.417564 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.417524    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2qcg\" (UniqueName: \"kubernetes.io/projected/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kube-api-access-v2qcg\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:06:39.417968 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.417579    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:06:39.417968 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.417591    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:06:39.417968 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.417601    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:06:39.417968 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.417610    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:06:39.417968 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.417618    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:06:39.452211 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.452168    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerID="7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb" exitCode=0
Sep 19 09:06:39.452406 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.452230    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"
Sep 19 09:06:39.452406 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.452237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" event={"ID":"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc","Type":"ContainerDied","Data":"7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb"}
Sep 19 09:06:39.452406 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.452296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs" event={"ID":"ee9e7cab-83d1-4fbe-b363-8a11db04e2bc","Type":"ContainerDied","Data":"bde42b0861c5b1bc30935acd5b31ea7d167c51f5d138bbbd05663b73c76dc7ff"}
Sep 19 09:06:39.452406 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.452326    2577 scope.go:117] "RemoveContainer" containerID="7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb"
Sep 19 09:06:39.462536 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.462514    2577 scope.go:117] "RemoveContainer" containerID="6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672"
Sep 19 09:06:39.470815 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.470794    2577 scope.go:117] "RemoveContainer" containerID="7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb"
Sep 19 09:06:39.471079 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:06:39.471058    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb\": container with ID starting with 7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb not found: ID does not exist" containerID="7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb"
Sep 19 09:06:39.471126 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.471089    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb"} err="failed to get container status \"7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb\": rpc error: code = NotFound desc = could not find container \"7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb\": container with ID starting with 7480e040c4bbcd6129ba94b8161d606d87a33470087a2c12d78fd16df46cc9cb not found: ID does not exist"
Sep 19 09:06:39.471126 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.471108    2577 scope.go:117] "RemoveContainer" containerID="6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672"
Sep 19 09:06:39.471340 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:06:39.471320    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672\": container with ID starting with 6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672 not found: ID does not exist" containerID="6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672"
Sep 19 09:06:39.471421 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.471346    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672"} err="failed to get container status \"6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672\": rpc error: code = NotFound desc = could not find container \"6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672\": container with ID starting with 6935ee0679a89570b9db9f2b4c299bd3f515fa98baf87ace0fe8c2d6acb53672 not found: ID does not exist"
Sep 19 09:06:39.475657 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.475635    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"]
Sep 19 09:06:39.478991 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:39.478964    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-g2pqs"]
Sep 19 09:06:40.504210 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:40.504179    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" path="/var/lib/kubelet/pods/ee9e7cab-83d1-4fbe-b363-8a11db04e2bc/volumes"
Sep 19 09:06:50.088972 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.088880    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"]
Sep 19 09:06:50.089606 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.089435    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="storage-initializer"
Sep 19 09:06:50.089606 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.089456    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="storage-initializer"
Sep 19 09:06:50.089606 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.089512    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main"
Sep 19 09:06:50.089606 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.089522    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main"
Sep 19 09:06:50.089841 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.089625    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee9e7cab-83d1-4fbe-b363-8a11db04e2bc" containerName="main"
Sep 19 09:06:50.094487 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.094458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.096977 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.096942    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 19 09:06:50.097107 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.096986    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 19 09:06:50.097202 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.097182    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 19 09:06:50.097274 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.097214    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 19 09:06:50.097501 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.097485    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:06:50.097575 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.097564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-85zwp\""
Sep 19 09:06:50.104797 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.104772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"]
Sep 19 09:06:50.207077 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.207031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/443785d1-4a26-4fab-93ba-9c7774b6fcdc-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.207390 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.207345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-home\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.207593 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.207575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.207748 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.207731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-dshm\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.207872 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.207856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.207982 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.207967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trljp\" (UniqueName: \"kubernetes.io/projected/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kube-api-access-trljp\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.208105 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.208088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.208236 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.208221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.309397 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.309332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.309581 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.309404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-dshm\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.309581 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.309455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.309581 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.309481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trljp\" (UniqueName: \"kubernetes.io/projected/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kube-api-access-trljp\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.309768 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.309598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.309768 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.309641    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.309768 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.309693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/443785d1-4a26-4fab-93ba-9c7774b6fcdc-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.309768 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.309728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-home\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.310221 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.310194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.310536 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.310510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.310671 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.310643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/443785d1-4a26-4fab-93ba-9c7774b6fcdc-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.310860 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.310839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.310989 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.310741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-home\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.312867 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.312841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.313079 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.313056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-dshm\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.319213 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.319178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trljp\" (UniqueName: \"kubernetes.io/projected/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kube-api-access-trljp\") pod \"stop-feature-test-kserve-6bfc568595-flmqr\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.406140 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.406109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:06:50.682489 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:50.682387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"]
Sep 19 09:06:50.684971 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:06:50.684937    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443785d1_4a26_4fab_93ba_9c7774b6fcdc.slice/crio-f45fb40aa64a695deab782a78da182dea1248c0c27888c508e19588cdd8637cd WatchSource:0}: Error finding container f45fb40aa64a695deab782a78da182dea1248c0c27888c508e19588cdd8637cd: Status 404 returned error can't find the container with id f45fb40aa64a695deab782a78da182dea1248c0c27888c508e19588cdd8637cd
Sep 19 09:06:51.519125 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:51.519088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" event={"ID":"443785d1-4a26-4fab-93ba-9c7774b6fcdc","Type":"ContainerStarted","Data":"8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7"}
Sep 19 09:06:51.519564 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:51.519141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" event={"ID":"443785d1-4a26-4fab-93ba-9c7774b6fcdc","Type":"ContainerStarted","Data":"f45fb40aa64a695deab782a78da182dea1248c0c27888c508e19588cdd8637cd"}
Sep 19 09:06:53.532772 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:53.532735    2577 generic.go:356] "Generic (PLEG): container finished" podID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerID="8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7" exitCode=0
Sep 19 09:06:53.533279 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:53.532793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" event={"ID":"443785d1-4a26-4fab-93ba-9c7774b6fcdc","Type":"ContainerDied","Data":"8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7"}
Sep 19 09:06:54.547574 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:54.547538    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" event={"ID":"443785d1-4a26-4fab-93ba-9c7774b6fcdc","Type":"ContainerStarted","Data":"93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716"}
Sep 19 09:06:54.571181 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:54.571118    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podStartSLOduration=2.436394032 podStartE2EDuration="4.571100569s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.134706537s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:06:54.568584633 +0000 UTC m=+2782.250168067" watchObservedRunningTime="2026-09-19 09:06:54.571100569 +0000 UTC m=+2782.252684001"
Sep 19 09:06:55.407271 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:06:55.407229    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:07:00.406674 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:07:00.406626    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:07:00.408444 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:07:00.408405    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:07:10.407053 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:07:10.407007    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:07:20.407185 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:07:20.407133    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:07:30.407211 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:07:30.407161    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:07:40.407537 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:07:40.407487    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:07:50.407031 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:07:50.406985    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:08:00.406726 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:08:00.406682    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:08:10.407543 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:08:10.407500    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:08:20.407419 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:08:20.407305    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:08:30.407325 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:08:30.407279    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:08:40.407388 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:08:40.407327    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:08:50.417724 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:08:50.417692    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:08:50.425675 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:08:50.425652    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:08:52.047993 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:08:52.047954    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"]
Sep 19 09:09:07.222229 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:07.222165    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" containerID="cri-o://93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716" gracePeriod=45
Sep 19 09:09:07.419256 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:07.419216    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 19 09:09:07.968642 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:07.968612    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:09:08.137220 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.137185    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tmp-dir\") pod \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") "
Sep 19 09:09:08.137460 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.137236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trljp\" (UniqueName: \"kubernetes.io/projected/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kube-api-access-trljp\") pod \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") "
Sep 19 09:09:08.137460 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.137267    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-home\") pod \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") "
Sep 19 09:09:08.137460 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.137301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-dshm\") pod \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") "
Sep 19 09:09:08.137460 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.137327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/443785d1-4a26-4fab-93ba-9c7774b6fcdc-cabundle-cert\") pod \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") "
Sep 19 09:09:08.137460 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.137393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-model-cache\") pod \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") "
Sep 19 09:09:08.137460 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.137431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tls-certs\") pod \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") "
Sep 19 09:09:08.137801 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.137486    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kserve-provision-location\") pod \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\" (UID: \"443785d1-4a26-4fab-93ba-9c7774b6fcdc\") "
Sep 19 09:09:08.137863 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.137827    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-model-cache" (OuterVolumeSpecName: "model-cache") pod "443785d1-4a26-4fab-93ba-9c7774b6fcdc" (UID: "443785d1-4a26-4fab-93ba-9c7774b6fcdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:08.138073 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.138044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443785d1-4a26-4fab-93ba-9c7774b6fcdc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "443785d1-4a26-4fab-93ba-9c7774b6fcdc" (UID: "443785d1-4a26-4fab-93ba-9c7774b6fcdc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:09:08.138205 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.138177    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-home" (OuterVolumeSpecName: "home") pod "443785d1-4a26-4fab-93ba-9c7774b6fcdc" (UID: "443785d1-4a26-4fab-93ba-9c7774b6fcdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:08.139748 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.139723    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "443785d1-4a26-4fab-93ba-9c7774b6fcdc" (UID: "443785d1-4a26-4fab-93ba-9c7774b6fcdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:09:08.139904 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.139885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-dshm" (OuterVolumeSpecName: "dshm") pod "443785d1-4a26-4fab-93ba-9c7774b6fcdc" (UID: "443785d1-4a26-4fab-93ba-9c7774b6fcdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:08.140039 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.140021    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kube-api-access-trljp" (OuterVolumeSpecName: "kube-api-access-trljp") pod "443785d1-4a26-4fab-93ba-9c7774b6fcdc" (UID: "443785d1-4a26-4fab-93ba-9c7774b6fcdc"). InnerVolumeSpecName "kube-api-access-trljp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:09:08.153184 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.153157    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "443785d1-4a26-4fab-93ba-9c7774b6fcdc" (UID: "443785d1-4a26-4fab-93ba-9c7774b6fcdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:08.162588 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.162566    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "443785d1-4a26-4fab-93ba-9c7774b6fcdc" (UID: "443785d1-4a26-4fab-93ba-9c7774b6fcdc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:08.238144 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.238110    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:09:08.238144 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.238141    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:09:08.238144 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.238150    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:09:08.238579 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.238160    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trljp\" (UniqueName: \"kubernetes.io/projected/443785d1-4a26-4fab-93ba-9c7774b6fcdc-kube-api-access-trljp\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:09:08.238579 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.238168    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:09:08.238579 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.238176    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:09:08.238579 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.238183    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/443785d1-4a26-4fab-93ba-9c7774b6fcdc-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:09:08.238579 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.238191    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/443785d1-4a26-4fab-93ba-9c7774b6fcdc-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:09:08.291020 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.290988    2577 generic.go:356] "Generic (PLEG): container finished" podID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerID="93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716" exitCode=0
Sep 19 09:09:08.291179 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.291041    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"
Sep 19 09:09:08.291179 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.291073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" event={"ID":"443785d1-4a26-4fab-93ba-9c7774b6fcdc","Type":"ContainerDied","Data":"93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716"}
Sep 19 09:09:08.291179 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.291128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr" event={"ID":"443785d1-4a26-4fab-93ba-9c7774b6fcdc","Type":"ContainerDied","Data":"f45fb40aa64a695deab782a78da182dea1248c0c27888c508e19588cdd8637cd"}
Sep 19 09:09:08.291179 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.291149    2577 scope.go:117] "RemoveContainer" containerID="93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716"
Sep 19 09:09:08.301022 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.301004    2577 scope.go:117] "RemoveContainer" containerID="8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7"
Sep 19 09:09:08.308830 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.308815    2577 scope.go:117] "RemoveContainer" containerID="93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716"
Sep 19 09:09:08.309094 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:09:08.309072    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716\": container with ID starting with 93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716 not found: ID does not exist" containerID="93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716"
Sep 19 09:09:08.309205 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.309100    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716"} err="failed to get container status \"93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716\": rpc error: code = NotFound desc = could not find container \"93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716\": container with ID starting with 93ec3ffc6f7c782f9e223d2d93cdd3be1adcc7b96828efc604aa7f72a8bf6716 not found: ID does not exist"
Sep 19 09:09:08.309205 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.309118    2577 scope.go:117] "RemoveContainer" containerID="8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7"
Sep 19 09:09:08.309336 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:09:08.309320    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7\": container with ID starting with 8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7 not found: ID does not exist" containerID="8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7"
Sep 19 09:09:08.309408 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.309341    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7"} err="failed to get container status \"8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7\": rpc error: code = NotFound desc = could not find container \"8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7\": container with ID starting with 8720e04b3015ae47977ccac6d2065d620704e6438b6aefba6ce79faad1dcfbf7 not found: ID does not exist"
Sep 19 09:09:08.315895 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.315873    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"]
Sep 19 09:09:08.319705 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.319683    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-flmqr"]
Sep 19 09:09:08.504712 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:08.504631    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" path="/var/lib/kubelet/pods/443785d1-4a26-4fab-93ba-9c7774b6fcdc/volumes"
Sep 19 09:09:15.735670 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.735631    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"]
Sep 19 09:09:15.737148 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.737117    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main"
Sep 19 09:09:15.737312 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.737300    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main"
Sep 19 09:09:15.737470 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.737458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="storage-initializer"
Sep 19 09:09:15.737556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.737546    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="storage-initializer"
Sep 19 09:09:15.737825 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.737812    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="443785d1-4a26-4fab-93ba-9c7774b6fcdc" containerName="main"
Sep 19 09:09:15.744166 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.744141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:15.747075 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.747042    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 19 09:09:15.747192 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.747177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 19 09:09:15.747358 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.747326    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-4xf8s\""
Sep 19 09:09:15.747443 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.747419    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-cpz7s\""
Sep 19 09:09:15.747506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.747436    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 19 09:09:15.747747 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.747725    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 19 09:09:15.747936 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.747913    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:09:15.757355 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.757331    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"]
Sep 19 09:09:15.906092 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.906051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:15.906284 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.906090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpljl\" (UniqueName: \"kubernetes.io/projected/f1899f5d-597d-42f0-9d8e-7557df887dcf-kube-api-access-qpljl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:15.906284 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.906158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:15.906284 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.906201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f5d-597d-42f0-9d8e-7557df887dcf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:15.906284 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.906263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:15.906565 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.906319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:15.906565 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.906335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1899f5d-597d-42f0-9d8e-7557df887dcf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:15.906565 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:15.906352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.007606 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.007520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpljl\" (UniqueName: \"kubernetes.io/projected/f1899f5d-597d-42f0-9d8e-7557df887dcf-kube-api-access-qpljl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.007606 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.007558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.007606 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.007580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f5d-597d-42f0-9d8e-7557df887dcf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.007928 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.007619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.007928 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.007740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.007928 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.007780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1899f5d-597d-42f0-9d8e-7557df887dcf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.007928 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.007815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.007928 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.007884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.008233 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.008125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.008233 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.008166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.008233 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.008213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.008422 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.008243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.008686 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.008663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f5d-597d-42f0-9d8e-7557df887dcf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.010129 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.010112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.010310 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.010294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1899f5d-597d-42f0-9d8e-7557df887dcf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.016226 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.016197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpljl\" (UniqueName: \"kubernetes.io/projected/f1899f5d-597d-42f0-9d8e-7557df887dcf-kube-api-access-qpljl\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.056844 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.056812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:16.342470 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.342438    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"]
Sep 19 09:09:16.343625 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:09:16.343596    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1899f5d_597d_42f0_9d8e_7557df887dcf.slice/crio-61cd020d8dd566021889efa48b91b50917ba7f6cb0064938ede2adf1b2b99e18 WatchSource:0}: Error finding container 61cd020d8dd566021889efa48b91b50917ba7f6cb0064938ede2adf1b2b99e18: Status 404 returned error can't find the container with id 61cd020d8dd566021889efa48b91b50917ba7f6cb0064938ede2adf1b2b99e18
Sep 19 09:09:16.345990 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:16.345973    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 09:09:17.344623 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:17.344589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" event={"ID":"f1899f5d-597d-42f0-9d8e-7557df887dcf","Type":"ContainerStarted","Data":"61cd020d8dd566021889efa48b91b50917ba7f6cb0064938ede2adf1b2b99e18"}
Sep 19 09:09:18.351000 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:18.350965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" event={"ID":"f1899f5d-597d-42f0-9d8e-7557df887dcf","Type":"ContainerStarted","Data":"2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224"}
Sep 19 09:09:18.351497 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:18.351037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:19.360941 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:19.360901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" event={"ID":"f1899f5d-597d-42f0-9d8e-7557df887dcf","Type":"ContainerStarted","Data":"17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108"}
Sep 19 09:09:21.374286 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:21.374251    2577 generic.go:356] "Generic (PLEG): container finished" podID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerID="17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108" exitCode=0
Sep 19 09:09:21.376294 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:21.374330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" event={"ID":"f1899f5d-597d-42f0-9d8e-7557df887dcf","Type":"ContainerDied","Data":"17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108"}
Sep 19 09:09:22.382695 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:22.382656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" event={"ID":"f1899f5d-597d-42f0-9d8e-7557df887dcf","Type":"ContainerStarted","Data":"7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880"}
Sep 19 09:09:22.419653 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:22.419584    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podStartSLOduration=3.843353108 podStartE2EDuration="7.419560124s" totalImagesPullingTime="1.324725465s" totalInitContainerRuntime="2.251481551s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:09:22.417729355 +0000 UTC m=+2930.099312793" watchObservedRunningTime="2026-09-19 09:09:22.419560124 +0000 UTC m=+2930.101143564"
Sep 19 09:09:26.057606 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:26.057560    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:26.057994 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:26.057639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:26.059008 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:26.058978    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:09:36.058228 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:36.058179    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:09:36.075060 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:36.075030    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:09:46.057804 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:46.057650    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:09:52.042424 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.042388    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"]
Sep 19 09:09:52.047406 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.047352    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.054446 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.054421    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:09:52.054446 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.054430    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 19 09:09:52.055536 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.055400    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-zj87f\""
Sep 19 09:09:52.055671 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.055653    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 19 09:09:52.055748 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.055697    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 19 09:09:52.055937 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.055921    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 19 09:09:52.077766 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.077740    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"]
Sep 19 09:09:52.118880 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.118845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65eb1f3-407e-456a-821b-f5b6b026ce5b-cabundle-cert\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.119033 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.118892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zwk6\" (UniqueName: \"kubernetes.io/projected/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kube-api-access-7zwk6\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.119033 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.118920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tmp-dir\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.119033 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.118975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.119033 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.119017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-dshm\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.119199 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.119087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tls-certs\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.119199 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.119104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-home\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.119199 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.119172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-model-cache\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220335 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65eb1f3-407e-456a-821b-f5b6b026ce5b-cabundle-cert\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220335 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwk6\" (UniqueName: \"kubernetes.io/projected/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kube-api-access-7zwk6\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220603 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tmp-dir\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220603 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220710 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-dshm\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220710 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tls-certs\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220818 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-home\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220818 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tmp-dir\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220818 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220772    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-model-cache\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.220963 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.220939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.221217 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.221190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65eb1f3-407e-456a-821b-f5b6b026ce5b-cabundle-cert\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.221319 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.221204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-home\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.221489 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.221467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-model-cache\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.223085 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.223061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-dshm\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.223592 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.223570    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tls-certs\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.234888 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.234860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwk6\" (UniqueName: \"kubernetes.io/projected/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kube-api-access-7zwk6\") pod \"lora-single-adapter-test-kserve-d754d949c-cxtvh\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.359478 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.359436    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:09:52.651826 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:52.651792    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"]
Sep 19 09:09:52.653956 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:09:52.653921    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65eb1f3_407e_456a_821b_f5b6b026ce5b.slice/crio-21cf822577f8dbbafe277e1f3f17d26e1bf5d40e36d94d429886a44e10eb49ae WatchSource:0}: Error finding container 21cf822577f8dbbafe277e1f3f17d26e1bf5d40e36d94d429886a44e10eb49ae: Status 404 returned error can't find the container with id 21cf822577f8dbbafe277e1f3f17d26e1bf5d40e36d94d429886a44e10eb49ae
Sep 19 09:09:53.573160 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:53.573126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" event={"ID":"e65eb1f3-407e-456a-821b-f5b6b026ce5b","Type":"ContainerStarted","Data":"744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786"}
Sep 19 09:09:53.573577 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:53.573176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" event={"ID":"e65eb1f3-407e-456a-821b-f5b6b026ce5b","Type":"ContainerStarted","Data":"21cf822577f8dbbafe277e1f3f17d26e1bf5d40e36d94d429886a44e10eb49ae"}
Sep 19 09:09:56.057956 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:56.057839    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:09:56.593240 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:56.593204    2577 generic.go:356] "Generic (PLEG): container finished" podID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerID="744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786" exitCode=0
Sep 19 09:09:56.593492 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:56.593269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" event={"ID":"e65eb1f3-407e-456a-821b-f5b6b026ce5b","Type":"ContainerDied","Data":"744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786"}
Sep 19 09:09:57.606576 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:57.606520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" event={"ID":"e65eb1f3-407e-456a-821b-f5b6b026ce5b","Type":"ContainerStarted","Data":"969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1"}
Sep 19 09:09:57.636191 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:57.636127    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podStartSLOduration=3.755369815 podStartE2EDuration="6.636107776s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.880737961s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:09:57.632119467 +0000 UTC m=+2965.313702903" watchObservedRunningTime="2026-09-19 09:09:57.636107776 +0000 UTC m=+2965.317691209"
Sep 19 09:09:58.360355 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:09:58.360307    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:10:02.360448 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:02.360406    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:10:02.362127 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:02.362095    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 19 09:10:06.058168 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:06.058117    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:10:12.360895 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:12.360840    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 19 09:10:16.057607 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:16.057567    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:10:22.360645 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:22.360595    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 19 09:10:26.058035 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:26.057983    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:10:32.360474 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:32.360425    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 19 09:10:34.767710 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:34.767667    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:10:34.775553 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:34.775529    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:10:36.058121 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:36.058065    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:10:42.360646 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:42.360596    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 19 09:10:46.058206 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:46.058151    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:10:52.360820 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:52.360777    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 19 09:10:56.058180 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:10:56.058130    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:11:02.360709 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:02.360656    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 19 09:11:06.058263 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:06.058211    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8001/health\": dial tcp 10.133.0.76:8001: connect: connection refused"
Sep 19 09:11:12.360562 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:12.360522    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 19 09:11:16.072733 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:16.072647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:11:16.084878 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:16.084851    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:11:22.369942 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:22.369899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:11:22.378087 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:22.378058    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:11:27.910835 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:27.910802    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"]
Sep 19 09:11:42.934374 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:42.934300    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main" containerID="cri-o://7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880" gracePeriod=45
Sep 19 09:11:43.746504 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:43.746452    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="llm-d-routing-sidecar" containerID="cri-o://2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224" gracePeriod=45
Sep 19 09:11:44.002236 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.002208    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:11:44.177994 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.177955    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-home\") pod \"f1899f5d-597d-42f0-9d8e-7557df887dcf\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") "
Sep 19 09:11:44.177994 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178004    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpljl\" (UniqueName: \"kubernetes.io/projected/f1899f5d-597d-42f0-9d8e-7557df887dcf-kube-api-access-qpljl\") pod \"f1899f5d-597d-42f0-9d8e-7557df887dcf\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") "
Sep 19 09:11:44.178248 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178048    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-kserve-provision-location\") pod \"f1899f5d-597d-42f0-9d8e-7557df887dcf\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") "
Sep 19 09:11:44.178248 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178067    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-tmp-dir\") pod \"f1899f5d-597d-42f0-9d8e-7557df887dcf\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") "
Sep 19 09:11:44.178248 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1899f5d-597d-42f0-9d8e-7557df887dcf-tls-certs\") pod \"f1899f5d-597d-42f0-9d8e-7557df887dcf\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") "
Sep 19 09:11:44.178248 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178137    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f5d-597d-42f0-9d8e-7557df887dcf-cabundle-cert\") pod \"f1899f5d-597d-42f0-9d8e-7557df887dcf\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") "
Sep 19 09:11:44.178248 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178158    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-model-cache\") pod \"f1899f5d-597d-42f0-9d8e-7557df887dcf\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") "
Sep 19 09:11:44.178248 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178189    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-dshm\") pod \"f1899f5d-597d-42f0-9d8e-7557df887dcf\" (UID: \"f1899f5d-597d-42f0-9d8e-7557df887dcf\") "
Sep 19 09:11:44.178636 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178484    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1899f5d-597d-42f0-9d8e-7557df887dcf" (UID: "f1899f5d-597d-42f0-9d8e-7557df887dcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:44.178823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178791    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-home" (OuterVolumeSpecName: "home") pod "f1899f5d-597d-42f0-9d8e-7557df887dcf" (UID: "f1899f5d-597d-42f0-9d8e-7557df887dcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:44.178923 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.178902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1899f5d-597d-42f0-9d8e-7557df887dcf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1899f5d-597d-42f0-9d8e-7557df887dcf" (UID: "f1899f5d-597d-42f0-9d8e-7557df887dcf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:11:44.180986 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.180960    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1899f5d-597d-42f0-9d8e-7557df887dcf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1899f5d-597d-42f0-9d8e-7557df887dcf" (UID: "f1899f5d-597d-42f0-9d8e-7557df887dcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:11:44.181095 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.180964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-dshm" (OuterVolumeSpecName: "dshm") pod "f1899f5d-597d-42f0-9d8e-7557df887dcf" (UID: "f1899f5d-597d-42f0-9d8e-7557df887dcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:44.182619 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.182594    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1899f5d-597d-42f0-9d8e-7557df887dcf-kube-api-access-qpljl" (OuterVolumeSpecName: "kube-api-access-qpljl") pod "f1899f5d-597d-42f0-9d8e-7557df887dcf" (UID: "f1899f5d-597d-42f0-9d8e-7557df887dcf"). InnerVolumeSpecName "kube-api-access-qpljl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:11:44.193521 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.193496    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1899f5d-597d-42f0-9d8e-7557df887dcf" (UID: "f1899f5d-597d-42f0-9d8e-7557df887dcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:44.210654 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.210629    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1899f5d-597d-42f0-9d8e-7557df887dcf" (UID: "f1899f5d-597d-42f0-9d8e-7557df887dcf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:44.266682 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.266592    2577 generic.go:356] "Generic (PLEG): container finished" podID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerID="7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880" exitCode=0
Sep 19 09:11:44.266682 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.266624    2577 generic.go:356] "Generic (PLEG): container finished" podID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerID="2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224" exitCode=0
Sep 19 09:11:44.266682 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.266658    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"
Sep 19 09:11:44.266881 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.266658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" event={"ID":"f1899f5d-597d-42f0-9d8e-7557df887dcf","Type":"ContainerDied","Data":"7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880"}
Sep 19 09:11:44.266881 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.266776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" event={"ID":"f1899f5d-597d-42f0-9d8e-7557df887dcf","Type":"ContainerDied","Data":"2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224"}
Sep 19 09:11:44.266881 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.266802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7" event={"ID":"f1899f5d-597d-42f0-9d8e-7557df887dcf","Type":"ContainerDied","Data":"61cd020d8dd566021889efa48b91b50917ba7f6cb0064938ede2adf1b2b99e18"}
Sep 19 09:11:44.266881 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.266831    2577 scope.go:117] "RemoveContainer" containerID="7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880"
Sep 19 09:11:44.276901 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.276882    2577 scope.go:117] "RemoveContainer" containerID="17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108"
Sep 19 09:11:44.279152 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.279130    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:11:44.279263 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.279154    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpljl\" (UniqueName: \"kubernetes.io/projected/f1899f5d-597d-42f0-9d8e-7557df887dcf-kube-api-access-qpljl\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:11:44.279263 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.279167    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:11:44.279263 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.279180    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:11:44.279263 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.279194    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1899f5d-597d-42f0-9d8e-7557df887dcf-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:11:44.279263 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.279206    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f5d-597d-42f0-9d8e-7557df887dcf-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:11:44.279263 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.279218    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:11:44.279263 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.279229    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1899f5d-597d-42f0-9d8e-7557df887dcf-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:11:44.285264 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.285246    2577 scope.go:117] "RemoveContainer" containerID="2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224"
Sep 19 09:11:44.293322 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.293300    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"]
Sep 19 09:11:44.293915 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.293894    2577 scope.go:117] "RemoveContainer" containerID="7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880"
Sep 19 09:11:44.294195 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:11:44.294172    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880\": container with ID starting with 7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880 not found: ID does not exist" containerID="7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880"
Sep 19 09:11:44.294262 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.294201    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880"} err="failed to get container status \"7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880\": rpc error: code = NotFound desc = could not find container \"7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880\": container with ID starting with 7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880 not found: ID does not exist"
Sep 19 09:11:44.294262 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.294220    2577 scope.go:117] "RemoveContainer" containerID="17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108"
Sep 19 09:11:44.294499 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:11:44.294483    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108\": container with ID starting with 17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108 not found: ID does not exist" containerID="17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108"
Sep 19 09:11:44.294550 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.294504    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108"} err="failed to get container status \"17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108\": rpc error: code = NotFound desc = could not find container \"17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108\": container with ID starting with 17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108 not found: ID does not exist"
Sep 19 09:11:44.294550 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.294517    2577 scope.go:117] "RemoveContainer" containerID="2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224"
Sep 19 09:11:44.294740 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:11:44.294721    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224\": container with ID starting with 2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224 not found: ID does not exist" containerID="2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224"
Sep 19 09:11:44.294781 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.294744    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224"} err="failed to get container status \"2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224\": rpc error: code = NotFound desc = could not find container \"2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224\": container with ID starting with 2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224 not found: ID does not exist"
Sep 19 09:11:44.294781 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.294760    2577 scope.go:117] "RemoveContainer" containerID="7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880"
Sep 19 09:11:44.294971 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.294955    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880"} err="failed to get container status \"7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880\": rpc error: code = NotFound desc = could not find container \"7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880\": container with ID starting with 7d17745bc8e793149a70556ae8d44e6ed405187e4a93d85cb5ed6d0a88de9880 not found: ID does not exist"
Sep 19 09:11:44.295018 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.294970    2577 scope.go:117] "RemoveContainer" containerID="17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108"
Sep 19 09:11:44.295192 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.295169    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108"} err="failed to get container status \"17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108\": rpc error: code = NotFound desc = could not find container \"17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108\": container with ID starting with 17ef99f9bbcb5ca81c90ff1e347abafc6e1ae25e50d274f806a3d6301f54f108 not found: ID does not exist"
Sep 19 09:11:44.295272 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.295194    2577 scope.go:117] "RemoveContainer" containerID="2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224"
Sep 19 09:11:44.295492 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.295473    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224"} err="failed to get container status \"2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224\": rpc error: code = NotFound desc = could not find container \"2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224\": container with ID starting with 2521bfbb121d636dc2dc5294526a726446594897c53918a9335c41a398133224 not found: ID does not exist"
Sep 19 09:11:44.300738 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.300714    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7hc6b7"]
Sep 19 09:11:44.510841 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:44.510807    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" path="/var/lib/kubelet/pods/f1899f5d-597d-42f0-9d8e-7557df887dcf/volumes"
Sep 19 09:11:49.570751 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:49.570724    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-d754d949c-cxtvh_e65eb1f3-407e-456a-821b-f5b6b026ce5b/storage-initializer/0.log"
Sep 19 09:11:49.580576 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:49.580555    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-d754d949c-cxtvh_e65eb1f3-407e-456a-821b-f5b6b026ce5b/main/0.log"
Sep 19 09:11:51.185100 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:11:51.185062    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"]
Sep 19 09:12:06.207015 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.206969    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" containerID="cri-o://969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1" gracePeriod=45
Sep 19 09:12:06.370697 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.370656    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 19 09:12:06.859564 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.859536    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:12:06.978399 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-home\") pod \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") "
Sep 19 09:12:06.978399 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978341    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tls-certs\") pod \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") "
Sep 19 09:12:06.978399 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tmp-dir\") pod \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") "
Sep 19 09:12:06.978681 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-model-cache\") pod \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") "
Sep 19 09:12:06.978681 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978482    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65eb1f3-407e-456a-821b-f5b6b026ce5b-cabundle-cert\") pod \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") "
Sep 19 09:12:06.978681 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978517    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zwk6\" (UniqueName: \"kubernetes.io/projected/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kube-api-access-7zwk6\") pod \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") "
Sep 19 09:12:06.978681 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kserve-provision-location\") pod \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") "
Sep 19 09:12:06.978681 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-dshm\") pod \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\" (UID: \"e65eb1f3-407e-456a-821b-f5b6b026ce5b\") "
Sep 19 09:12:06.978952 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978722    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e65eb1f3-407e-456a-821b-f5b6b026ce5b" (UID: "e65eb1f3-407e-456a-821b-f5b6b026ce5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:12:06.978952 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978876    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:12:06.978952 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978894    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-home" (OuterVolumeSpecName: "home") pod "e65eb1f3-407e-456a-821b-f5b6b026ce5b" (UID: "e65eb1f3-407e-456a-821b-f5b6b026ce5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:12:06.979086 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.978986    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65eb1f3-407e-456a-821b-f5b6b026ce5b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e65eb1f3-407e-456a-821b-f5b6b026ce5b" (UID: "e65eb1f3-407e-456a-821b-f5b6b026ce5b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:12:06.981101 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.981074    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-dshm" (OuterVolumeSpecName: "dshm") pod "e65eb1f3-407e-456a-821b-f5b6b026ce5b" (UID: "e65eb1f3-407e-456a-821b-f5b6b026ce5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:12:06.981101 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.981076    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e65eb1f3-407e-456a-821b-f5b6b026ce5b" (UID: "e65eb1f3-407e-456a-821b-f5b6b026ce5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:12:06.981254 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.981113    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kube-api-access-7zwk6" (OuterVolumeSpecName: "kube-api-access-7zwk6") pod "e65eb1f3-407e-456a-821b-f5b6b026ce5b" (UID: "e65eb1f3-407e-456a-821b-f5b6b026ce5b"). InnerVolumeSpecName "kube-api-access-7zwk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:12:06.998318 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:06.998283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e65eb1f3-407e-456a-821b-f5b6b026ce5b" (UID: "e65eb1f3-407e-456a-821b-f5b6b026ce5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:12:07.008003 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.007980    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e65eb1f3-407e-456a-821b-f5b6b026ce5b" (UID: "e65eb1f3-407e-456a-821b-f5b6b026ce5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:12:07.079497 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.079455    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:12:07.079497 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.079488    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:12:07.079497 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.079502    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65eb1f3-407e-456a-821b-f5b6b026ce5b-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:12:07.079728 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.079516    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zwk6\" (UniqueName: \"kubernetes.io/projected/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kube-api-access-7zwk6\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:12:07.079728 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.079527    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:12:07.079728 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.079540    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:12:07.079728 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.079550    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e65eb1f3-407e-456a-821b-f5b6b026ce5b-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:12:07.400163 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.400127    2577 generic.go:356] "Generic (PLEG): container finished" podID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerID="969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1" exitCode=0
Sep 19 09:12:07.400670 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.400188    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"
Sep 19 09:12:07.400670 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.400215    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" event={"ID":"e65eb1f3-407e-456a-821b-f5b6b026ce5b","Type":"ContainerDied","Data":"969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1"}
Sep 19 09:12:07.400670 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.400275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh" event={"ID":"e65eb1f3-407e-456a-821b-f5b6b026ce5b","Type":"ContainerDied","Data":"21cf822577f8dbbafe277e1f3f17d26e1bf5d40e36d94d429886a44e10eb49ae"}
Sep 19 09:12:07.400670 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.400299    2577 scope.go:117] "RemoveContainer" containerID="969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1"
Sep 19 09:12:07.410294 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.410276    2577 scope.go:117] "RemoveContainer" containerID="744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786"
Sep 19 09:12:07.418720 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.418703    2577 scope.go:117] "RemoveContainer" containerID="969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1"
Sep 19 09:12:07.418970 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:12:07.418951    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1\": container with ID starting with 969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1 not found: ID does not exist" containerID="969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1"
Sep 19 09:12:07.419041 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.418984    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1"} err="failed to get container status \"969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1\": rpc error: code = NotFound desc = could not find container \"969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1\": container with ID starting with 969a14c5b67a3f8743dc0f5af5d3cc57cd790cfb2a8de797725950ea1a52a1e1 not found: ID does not exist"
Sep 19 09:12:07.419041 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.419024    2577 scope.go:117] "RemoveContainer" containerID="744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786"
Sep 19 09:12:07.419263 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:12:07.419247    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786\": container with ID starting with 744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786 not found: ID does not exist" containerID="744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786"
Sep 19 09:12:07.419303 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.419268    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786"} err="failed to get container status \"744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786\": rpc error: code = NotFound desc = could not find container \"744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786\": container with ID starting with 744176ed45b65798e738051c91cdd82d468c2d9a595a9ea41b864035e918e786 not found: ID does not exist"
Sep 19 09:12:07.424982 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.424957    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"]
Sep 19 09:12:07.430237 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:07.430210    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-cxtvh"]
Sep 19 09:12:08.504293 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:08.504257    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" path="/var/lib/kubelet/pods/e65eb1f3-407e-456a-821b-f5b6b026ce5b/volumes"
Sep 19 09:12:22.291107 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291068    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"]
Sep 19 09:12:22.291548 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291479    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="storage-initializer"
Sep 19 09:12:22.291548 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291497    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="storage-initializer"
Sep 19 09:12:22.291548 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291507    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="storage-initializer"
Sep 19 09:12:22.291548 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291513    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="storage-initializer"
Sep 19 09:12:22.291548 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291530    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="llm-d-routing-sidecar"
Sep 19 09:12:22.291548 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291537    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="llm-d-routing-sidecar"
Sep 19 09:12:22.291548 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291545    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main"
Sep 19 09:12:22.291548 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291551    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main"
Sep 19 09:12:22.291823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291565    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main"
Sep 19 09:12:22.291823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291570    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main"
Sep 19 09:12:22.291823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291651    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="main"
Sep 19 09:12:22.291823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291659    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1899f5d-597d-42f0-9d8e-7557df887dcf" containerName="llm-d-routing-sidecar"
Sep 19 09:12:22.291823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.291668    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65eb1f3-407e-456a-821b-f5b6b026ce5b" containerName="main"
Sep 19 09:12:22.296412 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.296395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.299192 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.299162    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 19 09:12:22.299346 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.299213    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-x6glg\""
Sep 19 09:12:22.299346 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.299220    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 19 09:12:22.299346 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.299213    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 19 09:12:22.299346 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.299300    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:12:22.299346 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.299223    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 19 09:12:22.305879 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.305856    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"]
Sep 19 09:12:22.415495 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.415455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-model-cache\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.415495 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.415498    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.415713 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.415517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.415713 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.415556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5xhp\" (UniqueName: \"kubernetes.io/projected/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kube-api-access-v5xhp\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.415713 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.415598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-dshm\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.415713 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.415617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.415713 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.415654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tls-certs\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.415713 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.415672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-home\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516071 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516071 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516312 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5xhp\" (UniqueName: \"kubernetes.io/projected/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kube-api-access-v5xhp\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516312 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-dshm\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516312 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516312 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tls-certs\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516312 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-home\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516312 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-model-cache\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516636 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516636 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516729 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-model-cache\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516769 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-home\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.516901 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.516884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.518652 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.518631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-dshm\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.518839 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.518822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tls-certs\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.524314 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.524291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5xhp\" (UniqueName: \"kubernetes.io/projected/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kube-api-access-v5xhp\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.607399 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.607345    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:22.890484 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:22.890449    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"]
Sep 19 09:12:22.893144 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:12:22.893112    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5695da49_1245_4d83_9b47_b0ac7ca9b1f2.slice/crio-8b841f59c4d37484c7b308bd21d5de75196f4e710179c2e3e5c337febc07e07a WatchSource:0}: Error finding container 8b841f59c4d37484c7b308bd21d5de75196f4e710179c2e3e5c337febc07e07a: Status 404 returned error can't find the container with id 8b841f59c4d37484c7b308bd21d5de75196f4e710179c2e3e5c337febc07e07a
Sep 19 09:12:23.490329 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:23.490281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" event={"ID":"5695da49-1245-4d83-9b47-b0ac7ca9b1f2","Type":"ContainerStarted","Data":"5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc"}
Sep 19 09:12:23.490771 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:23.490353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" event={"ID":"5695da49-1245-4d83-9b47-b0ac7ca9b1f2","Type":"ContainerStarted","Data":"8b841f59c4d37484c7b308bd21d5de75196f4e710179c2e3e5c337febc07e07a"}
Sep 19 09:12:26.509570 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:26.509535    2577 generic.go:356] "Generic (PLEG): container finished" podID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerID="5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc" exitCode=0
Sep 19 09:12:26.509960 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:26.509587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" event={"ID":"5695da49-1245-4d83-9b47-b0ac7ca9b1f2","Type":"ContainerDied","Data":"5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc"}
Sep 19 09:12:27.518075 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:27.518036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" event={"ID":"5695da49-1245-4d83-9b47-b0ac7ca9b1f2","Type":"ContainerStarted","Data":"3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070"}
Sep 19 09:12:27.545219 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:27.545168    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podStartSLOduration=2.739116995 podStartE2EDuration="5.545151785s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.80603479s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:12:27.542711151 +0000 UTC m=+3115.224294575" watchObservedRunningTime="2026-09-19 09:12:27.545151785 +0000 UTC m=+3115.226735217"
Sep 19 09:12:27.607905 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:27.607869    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:32.608153 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:32.608109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:12:32.610073 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:32.610039    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 19 09:12:42.608249 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:42.608206    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 19 09:12:50.872178 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.872082    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"]
Sep 19 09:12:50.877999 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.877955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:50.881275 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.881245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 19 09:12:50.881459 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.881282    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:12:50.881459 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.881313    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 19 09:12:50.881459 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.881350    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-6kkx5\""
Sep 19 09:12:50.881459 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.881260    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 19 09:12:50.881459 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.881453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 19 09:12:50.890686 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.890656    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"]
Sep 19 09:12:50.978433 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.978385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:50.978614 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.978445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24f15e47-798a-46b9-84cf-d313361fb169-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:50.978614 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.978509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrcv6\" (UniqueName: \"kubernetes.io/projected/24f15e47-798a-46b9-84cf-d313361fb169-kube-api-access-wrcv6\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:50.978614 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.978587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:50.978985 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.978622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:50.978985 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.978665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:50.978985 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.978700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:50.978985 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:50.978729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f15e47-798a-46b9-84cf-d313361fb169-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.079378 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.079335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.079586 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.079414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f15e47-798a-46b9-84cf-d313361fb169-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.079586 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.079500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.079586 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.079527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24f15e47-798a-46b9-84cf-d313361fb169-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.079586 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.079570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrcv6\" (UniqueName: \"kubernetes.io/projected/24f15e47-798a-46b9-84cf-d313361fb169-kube-api-access-wrcv6\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.079784 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.079620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.079784 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.079650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.079784 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.079693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.080380 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.080142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.080380 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.080204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.080380 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.080302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.080380 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.080356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.080689 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.080437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24f15e47-798a-46b9-84cf-d313361fb169-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.082732 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.082705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.083120 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.083098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f15e47-798a-46b9-84cf-d313361fb169-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.096431 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.096402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrcv6\" (UniqueName: \"kubernetes.io/projected/24f15e47-798a-46b9-84cf-d313361fb169-kube-api-access-wrcv6\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.190186 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.190068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:12:51.483637 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.483599    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"]
Sep 19 09:12:51.484242 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:12:51.484213    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f15e47_798a_46b9_84cf_d313361fb169.slice/crio-bdcb9219b0c6cb47228fcb624d05b9a665cce3c129e90fdc99534edb8073da0a WatchSource:0}: Error finding container bdcb9219b0c6cb47228fcb624d05b9a665cce3c129e90fdc99534edb8073da0a: Status 404 returned error can't find the container with id bdcb9219b0c6cb47228fcb624d05b9a665cce3c129e90fdc99534edb8073da0a
Sep 19 09:12:51.651428 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.651356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" event={"ID":"24f15e47-798a-46b9-84cf-d313361fb169","Type":"ContainerStarted","Data":"87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0"}
Sep 19 09:12:51.651622 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:51.651459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" event={"ID":"24f15e47-798a-46b9-84cf-d313361fb169","Type":"ContainerStarted","Data":"bdcb9219b0c6cb47228fcb624d05b9a665cce3c129e90fdc99534edb8073da0a"}
Sep 19 09:12:52.608570 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:52.608517    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 19 09:12:54.670849 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:54.670813    2577 generic.go:356] "Generic (PLEG): container finished" podID="24f15e47-798a-46b9-84cf-d313361fb169" containerID="87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0" exitCode=0
Sep 19 09:12:54.671270 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:54.670862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" event={"ID":"24f15e47-798a-46b9-84cf-d313361fb169","Type":"ContainerDied","Data":"87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0"}
Sep 19 09:12:55.680059 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:55.680018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" event={"ID":"24f15e47-798a-46b9-84cf-d313361fb169","Type":"ContainerStarted","Data":"cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9"}
Sep 19 09:12:55.705566 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:12:55.705516    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podStartSLOduration=3.329610053 podStartE2EDuration="5.705499921s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.375889868s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:12:55.701675634 +0000 UTC m=+3143.383259069" watchObservedRunningTime="2026-09-19 09:12:55.705499921 +0000 UTC m=+3143.387083375"
Sep 19 09:13:01.191043 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:01.191007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:13:01.191538 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:01.191336    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:13:01.193026 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:01.192997    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:13:02.608398 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:02.608328    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 19 09:13:11.190924 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:11.190877    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:13:12.608953 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:12.608897    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 19 09:13:21.191355 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:21.191308    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:13:22.608262 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:22.608212    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 19 09:13:31.190790 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:31.190744    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:13:32.607785 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:32.607743    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 19 09:13:41.191310 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:41.191258    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:13:42.618480 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:42.618445    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:13:42.627085 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:42.627051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:13:51.190805 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:13:51.190744    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:14:01.190974 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:01.190910    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:14:10.794043 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:10.794002    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s_5695da49-1245-4d83-9b47-b0ac7ca9b1f2/storage-initializer/0.log"
Sep 19 09:14:10.803861 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:10.803834    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s_5695da49-1245-4d83-9b47-b0ac7ca9b1f2/main/0.log"
Sep 19 09:14:11.191413 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:11.191349    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:14:11.768580 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:11.768547    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"]
Sep 19 09:14:21.191338 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:21.191295    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:14:26.791488 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:26.791434    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main" containerID="cri-o://3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070" gracePeriod=45
Sep 19 09:14:27.619154 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.619107    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 19 09:14:27.757857 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.757824    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:14:27.901766 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.901722    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-cabundle-cert\") pod \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") "
Sep 19 09:14:27.902260 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.901789    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-home\") pod \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") "
Sep 19 09:14:27.902260 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.901908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kserve-provision-location\") pod \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") "
Sep 19 09:14:27.902260 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.901966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tls-certs\") pod \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") "
Sep 19 09:14:27.902260 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-model-cache\") pod \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") "
Sep 19 09:14:27.902260 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902063    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tmp-dir\") pod \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") "
Sep 19 09:14:27.902260 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902123    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-dshm\") pod \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") "
Sep 19 09:14:27.902260 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902151    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5xhp\" (UniqueName: \"kubernetes.io/projected/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kube-api-access-v5xhp\") pod \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\" (UID: \"5695da49-1245-4d83-9b47-b0ac7ca9b1f2\") "
Sep 19 09:14:27.902260 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902178    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5695da49-1245-4d83-9b47-b0ac7ca9b1f2" (UID: "5695da49-1245-4d83-9b47-b0ac7ca9b1f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:14:27.902260 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902225    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-home" (OuterVolumeSpecName: "home") pod "5695da49-1245-4d83-9b47-b0ac7ca9b1f2" (UID: "5695da49-1245-4d83-9b47-b0ac7ca9b1f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:14:27.902824 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5695da49-1245-4d83-9b47-b0ac7ca9b1f2" (UID: "5695da49-1245-4d83-9b47-b0ac7ca9b1f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:14:27.902824 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902584    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:14:27.902824 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902605    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:14:27.902824 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.902618    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:14:27.904918 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.904885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-dshm" (OuterVolumeSpecName: "dshm") pod "5695da49-1245-4d83-9b47-b0ac7ca9b1f2" (UID: "5695da49-1245-4d83-9b47-b0ac7ca9b1f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:14:27.904918 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.904908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kube-api-access-v5xhp" (OuterVolumeSpecName: "kube-api-access-v5xhp") pod "5695da49-1245-4d83-9b47-b0ac7ca9b1f2" (UID: "5695da49-1245-4d83-9b47-b0ac7ca9b1f2"). InnerVolumeSpecName "kube-api-access-v5xhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:14:27.905224 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.905194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5695da49-1245-4d83-9b47-b0ac7ca9b1f2" (UID: "5695da49-1245-4d83-9b47-b0ac7ca9b1f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:14:27.914848 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.914811    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5695da49-1245-4d83-9b47-b0ac7ca9b1f2" (UID: "5695da49-1245-4d83-9b47-b0ac7ca9b1f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:14:27.929426 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:27.929385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5695da49-1245-4d83-9b47-b0ac7ca9b1f2" (UID: "5695da49-1245-4d83-9b47-b0ac7ca9b1f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:14:28.003534 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.003489    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:14:28.003534 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.003520    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:14:28.003534 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.003535    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:14:28.003534 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.003547    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:14:28.003841 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.003556    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5xhp\" (UniqueName: \"kubernetes.io/projected/5695da49-1245-4d83-9b47-b0ac7ca9b1f2-kube-api-access-v5xhp\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:14:28.256645 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.256539    2577 generic.go:356] "Generic (PLEG): container finished" podID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerID="3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070" exitCode=0
Sep 19 09:14:28.256645 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.256601    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"
Sep 19 09:14:28.256645 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.256626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" event={"ID":"5695da49-1245-4d83-9b47-b0ac7ca9b1f2","Type":"ContainerDied","Data":"3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070"}
Sep 19 09:14:28.256922 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.256683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s" event={"ID":"5695da49-1245-4d83-9b47-b0ac7ca9b1f2","Type":"ContainerDied","Data":"8b841f59c4d37484c7b308bd21d5de75196f4e710179c2e3e5c337febc07e07a"}
Sep 19 09:14:28.256922 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.256712    2577 scope.go:117] "RemoveContainer" containerID="3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070"
Sep 19 09:14:28.267597 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.267571    2577 scope.go:117] "RemoveContainer" containerID="5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc"
Sep 19 09:14:28.276591 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.276560    2577 scope.go:117] "RemoveContainer" containerID="3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070"
Sep 19 09:14:28.276913 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:14:28.276893    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070\": container with ID starting with 3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070 not found: ID does not exist" containerID="3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070"
Sep 19 09:14:28.276962 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.276924    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070"} err="failed to get container status \"3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070\": rpc error: code = NotFound desc = could not find container \"3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070\": container with ID starting with 3eafbb2aac37f2c044de7148c7eb17486514b8d49fd03235bdc3f3a5d96ee070 not found: ID does not exist"
Sep 19 09:14:28.276962 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.276944    2577 scope.go:117] "RemoveContainer" containerID="5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc"
Sep 19 09:14:28.277227 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:14:28.277207    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc\": container with ID starting with 5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc not found: ID does not exist" containerID="5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc"
Sep 19 09:14:28.277290 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.277230    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc"} err="failed to get container status \"5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc\": rpc error: code = NotFound desc = could not find container \"5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc\": container with ID starting with 5a61544348e19b6db12549ac09c5ca92807e44a04db4417520fc901c9e077dfc not found: ID does not exist"
Sep 19 09:14:28.282505 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.282471    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"]
Sep 19 09:14:28.286173 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.286148    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-zhb9s"]
Sep 19 09:14:28.505071 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:28.505031    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" path="/var/lib/kubelet/pods/5695da49-1245-4d83-9b47-b0ac7ca9b1f2/volumes"
Sep 19 09:14:31.191577 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:31.191525    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:14:35.229521 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.229480    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"]
Sep 19 09:14:35.230057 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.230037    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main"
Sep 19 09:14:35.230102 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.230061    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main"
Sep 19 09:14:35.230138 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.230109    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="storage-initializer"
Sep 19 09:14:35.230138 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.230119    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="storage-initializer"
Sep 19 09:14:35.230208 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.230202    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5695da49-1245-4d83-9b47-b0ac7ca9b1f2" containerName="main"
Sep 19 09:14:35.235431 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.235410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:35.238138 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.238093    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 19 09:14:35.238994 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.238970    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"]
Sep 19 09:14:35.239111 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.239067    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-hcpqj\""
Sep 19 09:14:35.239186 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.239123    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 19 09:14:35.239244 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.239180    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 19 09:14:35.371135 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.371091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhmhs\" (UniqueName: \"kubernetes.io/projected/c7043bfb-ef3e-41ef-8565-184726dae5d3-kube-api-access-qhmhs\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:35.471785 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.471746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhmhs\" (UniqueName: \"kubernetes.io/projected/c7043bfb-ef3e-41ef-8565-184726dae5d3-kube-api-access-qhmhs\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:35.480750 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:35.480681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhmhs\" (UniqueName: \"kubernetes.io/projected/c7043bfb-ef3e-41ef-8565-184726dae5d3-kube-api-access-qhmhs\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:40.120889 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:40.120850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:40.135307 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:40.135272    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:40.222028 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:40.221984    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:40.222219 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:40.222141    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:40.226076 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:40.226053    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") DevicePath \"csi-8581e3e2fd57faccfcf1ea1c841d944960028fe5392b5e167ee0db4a6ffbbe15\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:40.444508 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:40.444472    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3c66b5c34abafc7bf28d8ff5e030726daa4637fb413c1eafd8841ad4dd9a8f77/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:40.458879 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:40.458847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"e2e-pvc-lora-download-4n57f\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:40.646782 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:40.646747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:40.930673 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:40.930642    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"]
Sep 19 09:14:40.932298 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:14:40.932267    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7043bfb_ef3e_41ef_8565_184726dae5d3.slice/crio-6964289c26c451239afdba7aa8b5810c95bd01a4dd725d4000adbf34d4968e4e WatchSource:0}: Error finding container 6964289c26c451239afdba7aa8b5810c95bd01a4dd725d4000adbf34d4968e4e: Status 404 returned error can't find the container with id 6964289c26c451239afdba7aa8b5810c95bd01a4dd725d4000adbf34d4968e4e
Sep 19 09:14:41.190867 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:41.190824    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": dial tcp 10.133.0.79:8000: connect: connection refused"
Sep 19 09:14:41.343536 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:41.343432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f" event={"ID":"c7043bfb-ef3e-41ef-8565-184726dae5d3","Type":"ContainerStarted","Data":"fb7da6382b808d2fac3478bb1c97912d2c6fa83faae0dd83b0570763911dd5eb"}
Sep 19 09:14:41.343536 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:41.343484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f" event={"ID":"c7043bfb-ef3e-41ef-8565-184726dae5d3","Type":"ContainerStarted","Data":"6964289c26c451239afdba7aa8b5810c95bd01a4dd725d4000adbf34d4968e4e"}
Sep 19 09:14:41.360138 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:41.360067    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f" podStartSLOduration=9.360046456 podStartE2EDuration="9.360046456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 09:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:14:41.358072508 +0000 UTC m=+3249.039655966" watchObservedRunningTime="2026-09-19 09:14:41.360046456 +0000 UTC m=+3249.041629890"
Sep 19 09:14:43.358226 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:43.358183    2577 generic.go:356] "Generic (PLEG): container finished" podID="c7043bfb-ef3e-41ef-8565-184726dae5d3" containerID="fb7da6382b808d2fac3478bb1c97912d2c6fa83faae0dd83b0570763911dd5eb" exitCode=0
Sep 19 09:14:43.358629 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:43.358248    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f" event={"ID":"c7043bfb-ef3e-41ef-8565-184726dae5d3","Type":"ContainerDied","Data":"fb7da6382b808d2fac3478bb1c97912d2c6fa83faae0dd83b0570763911dd5eb"}
Sep 19 09:14:44.523780 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:44.523752    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:44.664124 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:44.664032    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"c7043bfb-ef3e-41ef-8565-184726dae5d3\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") "
Sep 19 09:14:44.664288 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:44.664144    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhmhs\" (UniqueName: \"kubernetes.io/projected/c7043bfb-ef3e-41ef-8565-184726dae5d3-kube-api-access-qhmhs\") pod \"c7043bfb-ef3e-41ef-8565-184726dae5d3\" (UID: \"c7043bfb-ef3e-41ef-8565-184726dae5d3\") "
Sep 19 09:14:44.666471 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:44.666444    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7043bfb-ef3e-41ef-8565-184726dae5d3-kube-api-access-qhmhs" (OuterVolumeSpecName: "kube-api-access-qhmhs") pod "c7043bfb-ef3e-41ef-8565-184726dae5d3" (UID: "c7043bfb-ef3e-41ef-8565-184726dae5d3"). InnerVolumeSpecName "kube-api-access-qhmhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:14:44.668335 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:44.668315    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649" (OuterVolumeSpecName: "model-storage") pod "c7043bfb-ef3e-41ef-8565-184726dae5d3" (UID: "c7043bfb-ef3e-41ef-8565-184726dae5d3"). InnerVolumeSpecName "pvc-e0733508-915c-40d9-81b5-fc413fd79e5a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 19 09:14:44.765554 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:44.765510    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhmhs\" (UniqueName: \"kubernetes.io/projected/c7043bfb-ef3e-41ef-8565-184726dae5d3-kube-api-access-qhmhs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:14:44.765554 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:44.765561    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") on node \"ip-10-0-131-33.ec2.internal\" "
Sep 19 09:14:45.371192 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:45.371156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f" event={"ID":"c7043bfb-ef3e-41ef-8565-184726dae5d3","Type":"ContainerDied","Data":"6964289c26c451239afdba7aa8b5810c95bd01a4dd725d4000adbf34d4968e4e"}
Sep 19 09:14:45.371192 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:45.371197    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6964289c26c451239afdba7aa8b5810c95bd01a4dd725d4000adbf34d4968e4e"
Sep 19 09:14:45.371414 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:45.371173    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"
Sep 19 09:14:45.737286 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:45.737259    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-e0733508-915c-40d9-81b5-fc413fd79e5a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649") on node "ip-10-0-131-33.ec2.internal"
Sep 19 09:14:45.774395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:45.774351    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"csi-8581e3e2fd57faccfcf1ea1c841d944960028fe5392b5e167ee0db4a6ffbbe15\""
Sep 19 09:14:51.201350 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:51.201314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:14:51.209502 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:51.209471    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:14:57.886527 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.886491    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"]
Sep 19 09:14:57.887014 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.886864    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7043bfb-ef3e-41ef-8565-184726dae5d3" containerName="storage-initializer"
Sep 19 09:14:57.887014 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.886877    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7043bfb-ef3e-41ef-8565-184726dae5d3" containerName="storage-initializer"
Sep 19 09:14:57.887014 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.886967    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7043bfb-ef3e-41ef-8565-184726dae5d3" containerName="storage-initializer"
Sep 19 09:14:57.891603 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.891580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:57.895879 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.895856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 19 09:14:57.896008 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.895859    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 19 09:14:57.896081 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.895861    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-hcpqj\""
Sep 19 09:14:57.896081 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.895877    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 19 09:14:57.896081 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.896068    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:14:57.896251 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.895886    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 19 09:14:57.905595 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.905573    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"]
Sep 19 09:14:57.977938 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.977910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:57.978089 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.977943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-dshm\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:57.978089 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.977964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-home\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:57.978225 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.978084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:57.978225 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.978132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:57.978225 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.978170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:57.978385 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:57.978227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079121 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079121 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5mjr\" (UniqueName: \"kubernetes.io/projected/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kube-api-access-l5mjr\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079414 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079414 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079414 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079414 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079414 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-dshm\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079697 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-home\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079697 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079697 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079675    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079855 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079855 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-home\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.079940 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.079925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.081553 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.081533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-dshm\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.081875 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.081859    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.180882 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.180802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5mjr\" (UniqueName: \"kubernetes.io/projected/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kube-api-access-l5mjr\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:14:58.189568 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:14:58.189542    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5mjr\" (UniqueName: \"kubernetes.io/projected/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kube-api-access-l5mjr\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:00.599398 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:00.599346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:00.606706 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:00.606673    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:00.700553 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:00.700513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:00.700731 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:00.700648    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:00.703300 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:00.703277    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") DevicePath \"csi-8581e3e2fd57faccfcf1ea1c841d944960028fe5392b5e167ee0db4a6ffbbe15\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:00.788203 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:00.788174    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3c66b5c34abafc7bf28d8ff5e030726daa4637fb413c1eafd8841ad4dd9a8f77/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:00.807485 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:00.807455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:00.904227 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:00.904151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:01.184974 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:01.184936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"]
Sep 19 09:15:01.186040 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:15:01.186003    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7aab477_a58f_4c7c_a3d1_5aaaa4db782e.slice/crio-2194ae2afe2d873207d9e31548b8f9d8fcf19ba6751e46d76cd1b3c3b5b5ec60 WatchSource:0}: Error finding container 2194ae2afe2d873207d9e31548b8f9d8fcf19ba6751e46d76cd1b3c3b5b5ec60: Status 404 returned error can't find the container with id 2194ae2afe2d873207d9e31548b8f9d8fcf19ba6751e46d76cd1b3c3b5b5ec60
Sep 19 09:15:01.467253 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:01.467149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" event={"ID":"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e","Type":"ContainerStarted","Data":"5949dbf579ab65aef7fc3eab5c2caf35dd96fdcccbbb3c9cf1b3bb5115c9b3cb"}
Sep 19 09:15:01.467253 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:01.467205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" event={"ID":"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e","Type":"ContainerStarted","Data":"2194ae2afe2d873207d9e31548b8f9d8fcf19ba6751e46d76cd1b3c3b5b5ec60"}
Sep 19 09:15:03.478562 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:03.478520    2577 generic.go:356] "Generic (PLEG): container finished" podID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerID="5949dbf579ab65aef7fc3eab5c2caf35dd96fdcccbbb3c9cf1b3bb5115c9b3cb" exitCode=0
Sep 19 09:15:03.478938 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:03.478600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" event={"ID":"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e","Type":"ContainerDied","Data":"5949dbf579ab65aef7fc3eab5c2caf35dd96fdcccbbb3c9cf1b3bb5115c9b3cb"}
Sep 19 09:15:04.486044 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:04.486002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" event={"ID":"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e","Type":"ContainerStarted","Data":"cb613d2a790d41ab71e7d44f2c4cb9c994f4f9d9b122b14304a39d7329e22faa"}
Sep 19 09:15:04.514912 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:04.514850    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podStartSLOduration=5.34370588 podStartE2EDuration="7.51482824s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.17112236s" isStatefulPod=true podCreationTimestamp="2026-09-19 09:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:15:04.510543191 +0000 UTC m=+3272.192126625" watchObservedRunningTime="2026-09-19 09:15:04.51482824 +0000 UTC m=+3272.196411674"
Sep 19 09:15:04.905285 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:04.905237    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:10.361028 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:10.360990    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"]
Sep 19 09:15:10.904930 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:10.904892    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:15:10.906274 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:10.906239    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 19 09:15:20.904768 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:20.904717    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 19 09:15:25.390127 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:25.390079    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main" containerID="cri-o://cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9" gracePeriod=45
Sep 19 09:15:26.157262 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.157228    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:15:26.224907 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.224823    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-home\") pod \"24f15e47-798a-46b9-84cf-d313361fb169\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") "
Sep 19 09:15:26.224907 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.224864    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrcv6\" (UniqueName: \"kubernetes.io/projected/24f15e47-798a-46b9-84cf-d313361fb169-kube-api-access-wrcv6\") pod \"24f15e47-798a-46b9-84cf-d313361fb169\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") "
Sep 19 09:15:26.224907 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.224884    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f15e47-798a-46b9-84cf-d313361fb169-tls-certs\") pod \"24f15e47-798a-46b9-84cf-d313361fb169\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") "
Sep 19 09:15:26.225183 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.224910    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-dshm\") pod \"24f15e47-798a-46b9-84cf-d313361fb169\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") "
Sep 19 09:15:26.225183 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.224970    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24f15e47-798a-46b9-84cf-d313361fb169-cabundle-cert\") pod \"24f15e47-798a-46b9-84cf-d313361fb169\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") "
Sep 19 09:15:26.225183 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.224992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-model-cache\") pod \"24f15e47-798a-46b9-84cf-d313361fb169\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") "
Sep 19 09:15:26.225183 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.225017    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-kserve-provision-location\") pod \"24f15e47-798a-46b9-84cf-d313361fb169\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") "
Sep 19 09:15:26.225183 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.225065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-tmp-dir\") pod \"24f15e47-798a-46b9-84cf-d313361fb169\" (UID: \"24f15e47-798a-46b9-84cf-d313361fb169\") "
Sep 19 09:15:26.225499 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.225339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f15e47-798a-46b9-84cf-d313361fb169-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24f15e47-798a-46b9-84cf-d313361fb169" (UID: "24f15e47-798a-46b9-84cf-d313361fb169"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:15:26.225597 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.225565    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-model-cache" (OuterVolumeSpecName: "model-cache") pod "24f15e47-798a-46b9-84cf-d313361fb169" (UID: "24f15e47-798a-46b9-84cf-d313361fb169"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.225724 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.225695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-home" (OuterVolumeSpecName: "home") pod "24f15e47-798a-46b9-84cf-d313361fb169" (UID: "24f15e47-798a-46b9-84cf-d313361fb169"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.228162 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.228127    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-dshm" (OuterVolumeSpecName: "dshm") pod "24f15e47-798a-46b9-84cf-d313361fb169" (UID: "24f15e47-798a-46b9-84cf-d313361fb169"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.228162 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.228135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f15e47-798a-46b9-84cf-d313361fb169-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24f15e47-798a-46b9-84cf-d313361fb169" (UID: "24f15e47-798a-46b9-84cf-d313361fb169"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:15:26.228162 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.228145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f15e47-798a-46b9-84cf-d313361fb169-kube-api-access-wrcv6" (OuterVolumeSpecName: "kube-api-access-wrcv6") pod "24f15e47-798a-46b9-84cf-d313361fb169" (UID: "24f15e47-798a-46b9-84cf-d313361fb169"). InnerVolumeSpecName "kube-api-access-wrcv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:15:26.241712 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.241671    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24f15e47-798a-46b9-84cf-d313361fb169" (UID: "24f15e47-798a-46b9-84cf-d313361fb169"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.250797 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.250771    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24f15e47-798a-46b9-84cf-d313361fb169" (UID: "24f15e47-798a-46b9-84cf-d313361fb169"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.326185 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.326148    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.326185 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.326178    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.326185 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.326188    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrcv6\" (UniqueName: \"kubernetes.io/projected/24f15e47-798a-46b9-84cf-d313361fb169-kube-api-access-wrcv6\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.326437 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.326200    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f15e47-798a-46b9-84cf-d313361fb169-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.326437 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.326208    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.326437 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.326215    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24f15e47-798a-46b9-84cf-d313361fb169-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.326437 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.326223    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.326437 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.326231    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24f15e47-798a-46b9-84cf-d313361fb169-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.612506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.612465    2577 generic.go:356] "Generic (PLEG): container finished" podID="24f15e47-798a-46b9-84cf-d313361fb169" containerID="cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9" exitCode=0
Sep 19 09:15:26.613011 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.612527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" event={"ID":"24f15e47-798a-46b9-84cf-d313361fb169","Type":"ContainerDied","Data":"cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9"}
Sep 19 09:15:26.613011 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.612536    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"
Sep 19 09:15:26.613011 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.612577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df" event={"ID":"24f15e47-798a-46b9-84cf-d313361fb169","Type":"ContainerDied","Data":"bdcb9219b0c6cb47228fcb624d05b9a665cce3c129e90fdc99534edb8073da0a"}
Sep 19 09:15:26.613011 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.612602    2577 scope.go:117] "RemoveContainer" containerID="cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9"
Sep 19 09:15:26.623636 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.623609    2577 scope.go:117] "RemoveContainer" containerID="87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0"
Sep 19 09:15:26.632650 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.632631    2577 scope.go:117] "RemoveContainer" containerID="cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9"
Sep 19 09:15:26.632733 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.632652    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"]
Sep 19 09:15:26.632949 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:15:26.632924    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9\": container with ID starting with cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9 not found: ID does not exist" containerID="cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9"
Sep 19 09:15:26.633013 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.632957    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9"} err="failed to get container status \"cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9\": rpc error: code = NotFound desc = could not find container \"cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9\": container with ID starting with cedc06cba831a85b52acb263b0bf52cd8d7fb4697a33c9349476ce8840c8dac9 not found: ID does not exist"
Sep 19 09:15:26.633013 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.632975    2577 scope.go:117] "RemoveContainer" containerID="87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0"
Sep 19 09:15:26.633411 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:15:26.633382    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0\": container with ID starting with 87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0 not found: ID does not exist" containerID="87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0"
Sep 19 09:15:26.633498 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.633420    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0"} err="failed to get container status \"87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0\": rpc error: code = NotFound desc = could not find container \"87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0\": container with ID starting with 87db799bd8b3c70b09a5a2b371d51276837d48613ed9936934c7a355f9f45de0 not found: ID does not exist"
Sep 19 09:15:26.637106 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:26.637085    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-8w7df"]
Sep 19 09:15:28.503893 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:28.503861    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f15e47-798a-46b9-84cf-d313361fb169" path="/var/lib/kubelet/pods/24f15e47-798a-46b9-84cf-d313361fb169/volumes"
Sep 19 09:15:30.905303 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:30.905263    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 19 09:15:34.809252 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:34.809220    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:15:34.818644 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:34.818619    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:15:40.905485 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:40.905425    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 19 09:15:50.905323 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:15:50.905233    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 19 09:16:00.905313 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:00.905273    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 19 09:16:10.905206 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:10.905164    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 19 09:16:13.451008 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.450972    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"]
Sep 19 09:16:13.451513 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.451374    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main"
Sep 19 09:16:13.451513 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.451387    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main"
Sep 19 09:16:13.451513 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.451403    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="storage-initializer"
Sep 19 09:16:13.451513 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.451408    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="storage-initializer"
Sep 19 09:16:13.451513 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.451474    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f15e47-798a-46b9-84cf-d313361fb169" containerName="main"
Sep 19 09:16:13.454839 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.454816    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.462110 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.461563    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 19 09:16:13.462110 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.461579    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 19 09:16:13.462814 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.462780    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:16:13.462944 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.462925    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-pjczb\""
Sep 19 09:16:13.464054 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.464035    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 19 09:16:13.464211 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.464195    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 19 09:16:13.479624 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.479590    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"]
Sep 19 09:16:13.554841 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.554805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-home\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.555037 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.554867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/693cddac-c2d7-47ec-a062-c563f431a11b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.555037 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.554930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.555037 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.554957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.555037 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.554975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257gc\" (UniqueName: \"kubernetes.io/projected/693cddac-c2d7-47ec-a062-c563f431a11b-kube-api-access-257gc\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.555211 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.555046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693cddac-c2d7-47ec-a062-c563f431a11b-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.555211 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.555069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.555211 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.555093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.655910 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.655847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.655910 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.655914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656203 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.655961    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-home\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656203 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/693cddac-c2d7-47ec-a062-c563f431a11b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656203 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656203 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656203 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257gc\" (UniqueName: \"kubernetes.io/projected/693cddac-c2d7-47ec-a062-c563f431a11b-kube-api-access-257gc\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656203 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693cddac-c2d7-47ec-a062-c563f431a11b-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656572 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656651 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656651 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656766 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-home\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.656885 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.656861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/693cddac-c2d7-47ec-a062-c563f431a11b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.658451 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.658426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.658790 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.658772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693cddac-c2d7-47ec-a062-c563f431a11b-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.665578 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.665550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-257gc\" (UniqueName: \"kubernetes.io/projected/693cddac-c2d7-47ec-a062-c563f431a11b-kube-api-access-257gc\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-xzr48\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:13.766019 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:13.765857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:14.068003 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:14.067973    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"]
Sep 19 09:16:14.069010 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:16:14.068986    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693cddac_c2d7_47ec_a062_c563f431a11b.slice/crio-1801fea563fafbaedc710bf358605028f9f462e363dcd6b60125d3078ba09824 WatchSource:0}: Error finding container 1801fea563fafbaedc710bf358605028f9f462e363dcd6b60125d3078ba09824: Status 404 returned error can't find the container with id 1801fea563fafbaedc710bf358605028f9f462e363dcd6b60125d3078ba09824
Sep 19 09:16:14.883346 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:14.883304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" event={"ID":"693cddac-c2d7-47ec-a062-c563f431a11b","Type":"ContainerStarted","Data":"addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553"}
Sep 19 09:16:14.883834 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:14.883381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" event={"ID":"693cddac-c2d7-47ec-a062-c563f431a11b","Type":"ContainerStarted","Data":"1801fea563fafbaedc710bf358605028f9f462e363dcd6b60125d3078ba09824"}
Sep 19 09:16:16.897320 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:16.897284    2577 generic.go:356] "Generic (PLEG): container finished" podID="693cddac-c2d7-47ec-a062-c563f431a11b" containerID="addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553" exitCode=0
Sep 19 09:16:16.897734 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:16.897386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" event={"ID":"693cddac-c2d7-47ec-a062-c563f431a11b","Type":"ContainerDied","Data":"addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553"}
Sep 19 09:16:17.906940 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:17.906904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" event={"ID":"693cddac-c2d7-47ec-a062-c563f431a11b","Type":"ContainerStarted","Data":"350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4"}
Sep 19 09:16:17.932738 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:17.932673    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podStartSLOduration=2.329205769 podStartE2EDuration="4.932650595s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.603444826s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:16:17.929774458 +0000 UTC m=+3345.611357892" watchObservedRunningTime="2026-09-19 09:16:17.932650595 +0000 UTC m=+3345.614234029"
Sep 19 09:16:20.920182 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:20.920145    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:16:20.930402 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:20.930346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:16:23.766157 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:23.766118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:23.766577 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:23.766195    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:16:23.767976 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:23.767947    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:16:33.766324 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:33.766278    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:16:43.766951 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:43.766907    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:16:53.767315 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:16:53.767262    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:17:02.489748 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:02.489707    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72_f7aab477-a58f-4c7c-a3d1-5aaaa4db782e/storage-initializer/0.log"
Sep 19 09:17:02.499628 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:02.499562    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72_f7aab477-a58f-4c7c-a3d1-5aaaa4db782e/main/0.log"
Sep 19 09:17:03.626412 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:03.626348    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"]
Sep 19 09:17:03.766974 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:03.766932    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:17:13.766640 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:13.766597    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:17:18.649750 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:18.649647    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main" containerID="cri-o://cb613d2a790d41ab71e7d44f2c4cb9c994f4f9d9b122b14304a39d7329e22faa" gracePeriod=45
Sep 19 09:17:18.921413 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:18.921294    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 19 09:17:19.288992 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.288950    2577 generic.go:356] "Generic (PLEG): container finished" podID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerID="cb613d2a790d41ab71e7d44f2c4cb9c994f4f9d9b122b14304a39d7329e22faa" exitCode=0
Sep 19 09:17:19.289190 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.289020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" event={"ID":"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e","Type":"ContainerDied","Data":"cb613d2a790d41ab71e7d44f2c4cb9c994f4f9d9b122b14304a39d7329e22faa"}
Sep 19 09:17:19.311536 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.311506    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:17:19.369160 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369116    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-model-cache\") pod \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") "
Sep 19 09:17:19.369160 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369160    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kserve-provision-location\") pod \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") "
Sep 19 09:17:19.369470 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tls-certs\") pod \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") "
Sep 19 09:17:19.369470 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5mjr\" (UniqueName: \"kubernetes.io/projected/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kube-api-access-l5mjr\") pod \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") "
Sep 19 09:17:19.369470 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369285    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-home\") pod \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") "
Sep 19 09:17:19.369470 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369316    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tmp-dir\") pod \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") "
Sep 19 09:17:19.369470 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369433    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" (UID: "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:17:19.369810 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369785    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-home" (OuterVolumeSpecName: "home") pod "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" (UID: "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:17:19.369881 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-cabundle-cert\") pod \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") "
Sep 19 09:17:19.369944 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369911    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") pod \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") "
Sep 19 09:17:19.370010 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.369957    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-dshm\") pod \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\" (UID: \"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e\") "
Sep 19 09:17:19.370190 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.370164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" (UID: "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:17:19.370303 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.370287    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:17:19.370397 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.370308    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:17:19.370397 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.370322    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:17:19.372506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.372474    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kube-api-access-l5mjr" (OuterVolumeSpecName: "kube-api-access-l5mjr") pod "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" (UID: "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e"). InnerVolumeSpecName "kube-api-access-l5mjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:17:19.372862 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.372830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" (UID: "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:17:19.374199 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.374160    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-dshm" (OuterVolumeSpecName: "dshm") pod "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" (UID: "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:17:19.376132 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.376108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" (UID: "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e"). InnerVolumeSpecName "pvc-e0733508-915c-40d9-81b5-fc413fd79e5a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 19 09:17:19.383698 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.383668    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" (UID: "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:17:19.394748 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.394712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" (UID: "f7aab477-a58f-4c7c-a3d1-5aaaa4db782e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:17:19.471653 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.471561    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:17:19.471653 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.471614    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") on node \"ip-10-0-131-33.ec2.internal\" "
Sep 19 09:17:19.471653 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.471631    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:17:19.471653 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.471644    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:17:19.471653 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.471654    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:17:19.471934 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.471664    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5mjr\" (UniqueName: \"kubernetes.io/projected/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e-kube-api-access-l5mjr\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:17:19.482738 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.482712    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-e0733508-915c-40d9-81b5-fc413fd79e5a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649") on node "ip-10-0-131-33.ec2.internal"
Sep 19 09:17:19.572832 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:19.572784    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-e0733508-915c-40d9-81b5-fc413fd79e5a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09a66a22338e21649\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"csi-8581e3e2fd57faccfcf1ea1c841d944960028fe5392b5e167ee0db4a6ffbbe15\""
Sep 19 09:17:20.297349 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:20.297315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72" event={"ID":"f7aab477-a58f-4c7c-a3d1-5aaaa4db782e","Type":"ContainerDied","Data":"2194ae2afe2d873207d9e31548b8f9d8fcf19ba6751e46d76cd1b3c3b5b5ec60"}
Sep 19 09:17:20.297349 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:20.297340    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"
Sep 19 09:17:20.297893 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:20.297400    2577 scope.go:117] "RemoveContainer" containerID="cb613d2a790d41ab71e7d44f2c4cb9c994f4f9d9b122b14304a39d7329e22faa"
Sep 19 09:17:20.308030 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:20.308012    2577 scope.go:117] "RemoveContainer" containerID="5949dbf579ab65aef7fc3eab5c2caf35dd96fdcccbbb3c9cf1b3bb5115c9b3cb"
Sep 19 09:17:20.323022 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:20.322997    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"]
Sep 19 09:17:20.325164 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:20.325143    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-2mm72"]
Sep 19 09:17:20.505473 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:20.505382    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" path="/var/lib/kubelet/pods/f7aab477-a58f-4c7c-a3d1-5aaaa4db782e/volumes"
Sep 19 09:17:23.766494 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:23.766444    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:17:27.041727 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.041686    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"]
Sep 19 09:17:27.042234 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.042212    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="storage-initializer"
Sep 19 09:17:27.042282 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.042238    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="storage-initializer"
Sep 19 09:17:27.042282 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.042252    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main"
Sep 19 09:17:27.042282 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.042261    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main"
Sep 19 09:17:27.042407 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.042395    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7aab477-a58f-4c7c-a3d1-5aaaa4db782e" containerName="main"
Sep 19 09:17:27.046077 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.046058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:27.049006 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.048972    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-2j9ds\""
Sep 19 09:17:27.049773 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.049747    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 19 09:17:27.050129 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.049966    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 19 09:17:27.050129 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.049989    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 19 09:17:27.053323 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.053303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"]
Sep 19 09:17:27.143144 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.143104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dhf\" (UniqueName: \"kubernetes.io/projected/a266eaa1-09dd-4b72-8726-27225469cbb0-kube-api-access-p7dhf\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:27.244639 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.244601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dhf\" (UniqueName: \"kubernetes.io/projected/a266eaa1-09dd-4b72-8726-27225469cbb0-kube-api-access-p7dhf\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:27.254025 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:27.253997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dhf\" (UniqueName: \"kubernetes.io/projected/a266eaa1-09dd-4b72-8726-27225469cbb0-kube-api-access-p7dhf\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:28.692216 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:28.692176    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"]
Sep 19 09:17:28.696475 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:28.696447    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-4n57f"]
Sep 19 09:17:30.503674 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:30.503641    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7043bfb-ef3e-41ef-8565-184726dae5d3" path="/var/lib/kubelet/pods/c7043bfb-ef3e-41ef-8565-184726dae5d3/volumes"
Sep 19 09:17:33.400016 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:33.399976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:33.405846 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:33.405821    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:33.500731 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:33.500696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:33.500891 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:33.500825    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:33.504954 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:33.504935    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") DevicePath \"csi-c320ebc569d7b9ebdd42ef806151d9635f5043b75f1f3c3fb1ba2fa3869bb4bc\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:33.722782 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:33.722750    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fd7356b8fd12a54b8d910c5d7781c27aff52e33577231fc1e9e9d9b2fef69bb0/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:33.736397 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:33.736348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"e2e-pvc-lora-download-8hw5p\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:33.766687 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:33.766643    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:17:33.956551 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:33.956516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:34.228059 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:34.228031    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"]
Sep 19 09:17:34.229987 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:17:34.229953    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda266eaa1_09dd_4b72_8726_27225469cbb0.slice/crio-76154cb93ac7baf2b553a69a0189f2e5eaa9ee8d0a317dfe64545bf1988d0aa8 WatchSource:0}: Error finding container 76154cb93ac7baf2b553a69a0189f2e5eaa9ee8d0a317dfe64545bf1988d0aa8: Status 404 returned error can't find the container with id 76154cb93ac7baf2b553a69a0189f2e5eaa9ee8d0a317dfe64545bf1988d0aa8
Sep 19 09:17:34.379825 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:34.379785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p" event={"ID":"a266eaa1-09dd-4b72-8726-27225469cbb0","Type":"ContainerStarted","Data":"d2eae3bb330871891de919fa977114eb05f3e33e1abaee36d906789f6d1ae4b4"}
Sep 19 09:17:34.380012 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:34.379840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p" event={"ID":"a266eaa1-09dd-4b72-8726-27225469cbb0","Type":"ContainerStarted","Data":"76154cb93ac7baf2b553a69a0189f2e5eaa9ee8d0a317dfe64545bf1988d0aa8"}
Sep 19 09:17:34.397606 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:34.397545    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p" podStartSLOduration=10.397523538 podStartE2EDuration="10.397523538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 09:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:17:34.394224785 +0000 UTC m=+3422.075808218" watchObservedRunningTime="2026-09-19 09:17:34.397523538 +0000 UTC m=+3422.079106972"
Sep 19 09:17:36.394160 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:36.394126    2577 generic.go:356] "Generic (PLEG): container finished" podID="a266eaa1-09dd-4b72-8726-27225469cbb0" containerID="d2eae3bb330871891de919fa977114eb05f3e33e1abaee36d906789f6d1ae4b4" exitCode=0
Sep 19 09:17:36.394554 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:36.394194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p" event={"ID":"a266eaa1-09dd-4b72-8726-27225469cbb0","Type":"ContainerDied","Data":"d2eae3bb330871891de919fa977114eb05f3e33e1abaee36d906789f6d1ae4b4"}
Sep 19 09:17:37.529278 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:37.529250    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:37.641706 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:37.641663    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7dhf\" (UniqueName: \"kubernetes.io/projected/a266eaa1-09dd-4b72-8726-27225469cbb0-kube-api-access-p7dhf\") pod \"a266eaa1-09dd-4b72-8726-27225469cbb0\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") "
Sep 19 09:17:37.641902 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:37.641867    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"a266eaa1-09dd-4b72-8726-27225469cbb0\" (UID: \"a266eaa1-09dd-4b72-8726-27225469cbb0\") "
Sep 19 09:17:37.644187 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:37.644155    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a266eaa1-09dd-4b72-8726-27225469cbb0-kube-api-access-p7dhf" (OuterVolumeSpecName: "kube-api-access-p7dhf") pod "a266eaa1-09dd-4b72-8726-27225469cbb0" (UID: "a266eaa1-09dd-4b72-8726-27225469cbb0"). InnerVolumeSpecName "kube-api-access-p7dhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:17:37.644811 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:37.644787    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439" (OuterVolumeSpecName: "model-storage") pod "a266eaa1-09dd-4b72-8726-27225469cbb0" (UID: "a266eaa1-09dd-4b72-8726-27225469cbb0"). InnerVolumeSpecName "pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 19 09:17:37.742754 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:37.742718    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7dhf\" (UniqueName: \"kubernetes.io/projected/a266eaa1-09dd-4b72-8726-27225469cbb0-kube-api-access-p7dhf\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:17:37.742934 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:37.742770    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") on node \"ip-10-0-131-33.ec2.internal\" "
Sep 19 09:17:38.406614 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:38.406585    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"
Sep 19 09:17:38.406795 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:38.406616    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p" event={"ID":"a266eaa1-09dd-4b72-8726-27225469cbb0","Type":"ContainerDied","Data":"76154cb93ac7baf2b553a69a0189f2e5eaa9ee8d0a317dfe64545bf1988d0aa8"}
Sep 19 09:17:38.406795 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:38.406651    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76154cb93ac7baf2b553a69a0189f2e5eaa9ee8d0a317dfe64545bf1988d0aa8"
Sep 19 09:17:38.713559 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:38.713534    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439") on node "ip-10-0-131-33.ec2.internal"
Sep 19 09:17:38.751907 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:38.751865    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"csi-c320ebc569d7b9ebdd42ef806151d9635f5043b75f1f3c3fb1ba2fa3869bb4bc\""
Sep 19 09:17:43.766350 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:43.766300    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:17:47.066142 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.066107    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"]
Sep 19 09:17:47.066557 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.066523    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a266eaa1-09dd-4b72-8726-27225469cbb0" containerName="storage-initializer"
Sep 19 09:17:47.066557 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.066544    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a266eaa1-09dd-4b72-8726-27225469cbb0" containerName="storage-initializer"
Sep 19 09:17:47.066654 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.066642    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a266eaa1-09dd-4b72-8726-27225469cbb0" containerName="storage-initializer"
Sep 19 09:17:47.069820 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.069801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.072603 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.072577    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 19 09:17:47.072755 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.072669    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 19 09:17:47.072755 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.072693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-2j9ds\""
Sep 19 09:17:47.072755 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.072731    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 19 09:17:47.082254 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.082225    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"]
Sep 19 09:17:47.224604 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.224564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.224794 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.224624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.224794 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.224673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.224794 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.224693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744f29b-e303-4c34-a559-f13c4593d522-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.224794 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.224769    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glxrx\" (UniqueName: \"kubernetes.io/projected/f744f29b-e303-4c34-a559-f13c4593d522-kube-api-access-glxrx\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.224948 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.224804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.325448 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.325340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glxrx\" (UniqueName: \"kubernetes.io/projected/f744f29b-e303-4c34-a559-f13c4593d522-kube-api-access-glxrx\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.325448 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.325417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.325660 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.325469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.325660 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.325525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.325660 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.325564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.325660 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.325595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744f29b-e303-4c34-a559-f13c4593d522-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.325905 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.325878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.325977 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.325948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.326040 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.326002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.327799 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.327776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.328075 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.328058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744f29b-e303-4c34-a559-f13c4593d522-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:47.333132 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:47.333108    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glxrx\" (UniqueName: \"kubernetes.io/projected/f744f29b-e303-4c34-a559-f13c4593d522-kube-api-access-glxrx\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:53.684874 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:53.684833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:53.690677 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:53.690649    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:53.766713 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:53.766671    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:17:53.785970 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:53.785940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:53.786095 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:53.786058    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:53.789272 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:53.789241    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") DevicePath \"csi-c320ebc569d7b9ebdd42ef806151d9635f5043b75f1f3c3fb1ba2fa3869bb4bc\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:53.876070 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:53.876032    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fd7356b8fd12a54b8d910c5d7781c27aff52e33577231fc1e9e9d9b2fef69bb0/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:53.897466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:53.897436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:53.982434 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:53.982322    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:17:54.263438 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:54.263406    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"]
Sep 19 09:17:54.265815 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:17:54.265783    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf744f29b_e303_4c34_a559_f13c4593d522.slice/crio-9ef131dec727f66889e55bbc30d2370cbada6c1083e9814a53208b96b69d8615 WatchSource:0}: Error finding container 9ef131dec727f66889e55bbc30d2370cbada6c1083e9814a53208b96b69d8615: Status 404 returned error can't find the container with id 9ef131dec727f66889e55bbc30d2370cbada6c1083e9814a53208b96b69d8615
Sep 19 09:17:54.506119 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:54.506078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" event={"ID":"f744f29b-e303-4c34-a559-f13c4593d522","Type":"ContainerStarted","Data":"8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13"}
Sep 19 09:17:54.506296 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:54.506132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" event={"ID":"f744f29b-e303-4c34-a559-f13c4593d522","Type":"ContainerStarted","Data":"9ef131dec727f66889e55bbc30d2370cbada6c1083e9814a53208b96b69d8615"}
Sep 19 09:17:54.526713 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:54.526586    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podStartSLOduration=7.526564604 podStartE2EDuration="7.526564604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 09:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:17:54.52171527 +0000 UTC m=+3442.203298703" watchObservedRunningTime="2026-09-19 09:17:54.526564604 +0000 UTC m=+3442.208148038"
Sep 19 09:17:54.983401 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:17:54.983335    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:18:03.766939 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:03.766884    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": dial tcp 10.133.0.82:8000: connect: connection refused"
Sep 19 09:18:03.983557 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:03.983511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:18:03.985380 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:03.985336    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 19 09:18:13.777542 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:13.777507    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:18:13.785276 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:13.785250    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:18:13.983769 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:13.983724    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 19 09:18:23.260286 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:23.260249    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"]
Sep 19 09:18:23.983181 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:23.983125    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 19 09:18:33.982898 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:33.982853    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 19 09:18:38.282953 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:38.282890    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main" containerID="cri-o://350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4" gracePeriod=45
Sep 19 09:18:39.044345 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.044314    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:18:39.098202 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.098101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-kserve-provision-location\") pod \"693cddac-c2d7-47ec-a062-c563f431a11b\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") "
Sep 19 09:18:39.098202 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.098179    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-home\") pod \"693cddac-c2d7-47ec-a062-c563f431a11b\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") "
Sep 19 09:18:39.098496 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.098216    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-tmp-dir\") pod \"693cddac-c2d7-47ec-a062-c563f431a11b\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") "
Sep 19 09:18:39.098496 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.098270    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693cddac-c2d7-47ec-a062-c563f431a11b-tls-certs\") pod \"693cddac-c2d7-47ec-a062-c563f431a11b\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") "
Sep 19 09:18:39.098496 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.098301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-dshm\") pod \"693cddac-c2d7-47ec-a062-c563f431a11b\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") "
Sep 19 09:18:39.098496 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.098342    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-model-cache\") pod \"693cddac-c2d7-47ec-a062-c563f431a11b\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") "
Sep 19 09:18:39.098496 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.098453    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-257gc\" (UniqueName: \"kubernetes.io/projected/693cddac-c2d7-47ec-a062-c563f431a11b-kube-api-access-257gc\") pod \"693cddac-c2d7-47ec-a062-c563f431a11b\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") "
Sep 19 09:18:39.098496 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.098487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/693cddac-c2d7-47ec-a062-c563f431a11b-cabundle-cert\") pod \"693cddac-c2d7-47ec-a062-c563f431a11b\" (UID: \"693cddac-c2d7-47ec-a062-c563f431a11b\") "
Sep 19 09:18:39.099133 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.098868    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-home" (OuterVolumeSpecName: "home") pod "693cddac-c2d7-47ec-a062-c563f431a11b" (UID: "693cddac-c2d7-47ec-a062-c563f431a11b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.099133 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.099091    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-model-cache" (OuterVolumeSpecName: "model-cache") pod "693cddac-c2d7-47ec-a062-c563f431a11b" (UID: "693cddac-c2d7-47ec-a062-c563f431a11b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.099300 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.099166    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693cddac-c2d7-47ec-a062-c563f431a11b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "693cddac-c2d7-47ec-a062-c563f431a11b" (UID: "693cddac-c2d7-47ec-a062-c563f431a11b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:18:39.101118 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.101086    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-dshm" (OuterVolumeSpecName: "dshm") pod "693cddac-c2d7-47ec-a062-c563f431a11b" (UID: "693cddac-c2d7-47ec-a062-c563f431a11b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.101416 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.101358    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693cddac-c2d7-47ec-a062-c563f431a11b-kube-api-access-257gc" (OuterVolumeSpecName: "kube-api-access-257gc") pod "693cddac-c2d7-47ec-a062-c563f431a11b" (UID: "693cddac-c2d7-47ec-a062-c563f431a11b"). InnerVolumeSpecName "kube-api-access-257gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:18:39.101416 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.101406    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693cddac-c2d7-47ec-a062-c563f431a11b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "693cddac-c2d7-47ec-a062-c563f431a11b" (UID: "693cddac-c2d7-47ec-a062-c563f431a11b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:18:39.113073 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.113043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "693cddac-c2d7-47ec-a062-c563f431a11b" (UID: "693cddac-c2d7-47ec-a062-c563f431a11b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.122207 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.122167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "693cddac-c2d7-47ec-a062-c563f431a11b" (UID: "693cddac-c2d7-47ec-a062-c563f431a11b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.199998 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.199958    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693cddac-c2d7-47ec-a062-c563f431a11b-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.199998 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.199990    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.199998 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.200001    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.199998 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.200010    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257gc\" (UniqueName: \"kubernetes.io/projected/693cddac-c2d7-47ec-a062-c563f431a11b-kube-api-access-257gc\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.200395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.200019    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/693cddac-c2d7-47ec-a062-c563f431a11b-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.200395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.200028    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.200395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.200036    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.200395 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.200043    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693cddac-c2d7-47ec-a062-c563f431a11b-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.783573 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.783539    2577 generic.go:356] "Generic (PLEG): container finished" podID="693cddac-c2d7-47ec-a062-c563f431a11b" containerID="350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4" exitCode=0
Sep 19 09:18:39.784000 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.783610    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"
Sep 19 09:18:39.784000 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.783631    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" event={"ID":"693cddac-c2d7-47ec-a062-c563f431a11b","Type":"ContainerDied","Data":"350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4"}
Sep 19 09:18:39.784000 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.783705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48" event={"ID":"693cddac-c2d7-47ec-a062-c563f431a11b","Type":"ContainerDied","Data":"1801fea563fafbaedc710bf358605028f9f462e363dcd6b60125d3078ba09824"}
Sep 19 09:18:39.784000 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.783740    2577 scope.go:117] "RemoveContainer" containerID="350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4"
Sep 19 09:18:39.794387 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.794349    2577 scope.go:117] "RemoveContainer" containerID="addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553"
Sep 19 09:18:39.803971 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.803946    2577 scope.go:117] "RemoveContainer" containerID="350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4"
Sep 19 09:18:39.804466 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:18:39.804430    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4\": container with ID starting with 350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4 not found: ID does not exist" containerID="350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4"
Sep 19 09:18:39.804623 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.804479    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4"} err="failed to get container status \"350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4\": rpc error: code = NotFound desc = could not find container \"350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4\": container with ID starting with 350a6a8360ea6d8cddcaad5a5d3a2b15980f52dba68394472c85963a99193ea4 not found: ID does not exist"
Sep 19 09:18:39.804623 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.804507    2577 scope.go:117] "RemoveContainer" containerID="addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553"
Sep 19 09:18:39.804929 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:18:39.804858    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553\": container with ID starting with addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553 not found: ID does not exist" containerID="addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553"
Sep 19 09:18:39.804929 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.804894    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553"} err="failed to get container status \"addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553\": rpc error: code = NotFound desc = could not find container \"addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553\": container with ID starting with addc7551d8db5d1258722b7ae005d31934dab35ef54c44fc8f3abe1a08820553 not found: ID does not exist"
Sep 19 09:18:39.807761 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.807732    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"]
Sep 19 09:18:39.811569 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:39.811548    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-xzr48"]
Sep 19 09:18:40.504328 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:40.504287    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" path="/var/lib/kubelet/pods/693cddac-c2d7-47ec-a062-c563f431a11b/volumes"
Sep 19 09:18:43.983410 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:43.983345    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 19 09:18:53.983328 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:18:53.983232    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 19 09:19:03.982847 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:19:03.982795    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": dial tcp 10.133.0.84:8000: connect: connection refused"
Sep 19 09:19:13.996377 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:19:13.996340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:19:14.003964 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:19:14.003936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:19:55.076140 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:19:55.076105    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf_f744f29b-e303-4c34-a559-f13c4593d522/main/0.log"
Sep 19 09:19:56.372649 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:19:56.372602    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"]
Sep 19 09:20:11.395718 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:11.395679    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main" containerID="cri-o://8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13" gracePeriod=45
Sep 19 09:20:12.056507 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.056476    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:20:12.116901 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.116802    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") pod \"f744f29b-e303-4c34-a559-f13c4593d522\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") "
Sep 19 09:20:12.116901 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.116860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-dshm\") pod \"f744f29b-e303-4c34-a559-f13c4593d522\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") "
Sep 19 09:20:12.116901 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.116879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-home\") pod \"f744f29b-e303-4c34-a559-f13c4593d522\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") "
Sep 19 09:20:12.117197 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.116905    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-model-cache\") pod \"f744f29b-e303-4c34-a559-f13c4593d522\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") "
Sep 19 09:20:12.117197 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.116928    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glxrx\" (UniqueName: \"kubernetes.io/projected/f744f29b-e303-4c34-a559-f13c4593d522-kube-api-access-glxrx\") pod \"f744f29b-e303-4c34-a559-f13c4593d522\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") "
Sep 19 09:20:12.117197 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.116960    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744f29b-e303-4c34-a559-f13c4593d522-tls-certs\") pod \"f744f29b-e303-4c34-a559-f13c4593d522\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") "
Sep 19 09:20:12.117197 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.116984    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-tmp-dir\") pod \"f744f29b-e303-4c34-a559-f13c4593d522\" (UID: \"f744f29b-e303-4c34-a559-f13c4593d522\") "
Sep 19 09:20:12.117427 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.117206    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-model-cache" (OuterVolumeSpecName: "model-cache") pod "f744f29b-e303-4c34-a559-f13c4593d522" (UID: "f744f29b-e303-4c34-a559-f13c4593d522"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:20:12.117427 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.117319    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-home" (OuterVolumeSpecName: "home") pod "f744f29b-e303-4c34-a559-f13c4593d522" (UID: "f744f29b-e303-4c34-a559-f13c4593d522"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:20:12.119177 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.119149    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-dshm" (OuterVolumeSpecName: "dshm") pod "f744f29b-e303-4c34-a559-f13c4593d522" (UID: "f744f29b-e303-4c34-a559-f13c4593d522"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:20:12.119306 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.119187    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f744f29b-e303-4c34-a559-f13c4593d522-kube-api-access-glxrx" (OuterVolumeSpecName: "kube-api-access-glxrx") pod "f744f29b-e303-4c34-a559-f13c4593d522" (UID: "f744f29b-e303-4c34-a559-f13c4593d522"). InnerVolumeSpecName "kube-api-access-glxrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:20:12.119355 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.119325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f744f29b-e303-4c34-a559-f13c4593d522-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f744f29b-e303-4c34-a559-f13c4593d522" (UID: "f744f29b-e303-4c34-a559-f13c4593d522"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:20:12.124890 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.124866    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439" (OuterVolumeSpecName: "kserve-pvc-source") pod "f744f29b-e303-4c34-a559-f13c4593d522" (UID: "f744f29b-e303-4c34-a559-f13c4593d522"). InnerVolumeSpecName "pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 19 09:20:12.130544 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.130498    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f744f29b-e303-4c34-a559-f13c4593d522" (UID: "f744f29b-e303-4c34-a559-f13c4593d522"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:20:12.218346 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.218311    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:20:12.218346 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.218344    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:20:12.218659 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.218355    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:20:12.218659 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.218401    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glxrx\" (UniqueName: \"kubernetes.io/projected/f744f29b-e303-4c34-a559-f13c4593d522-kube-api-access-glxrx\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:20:12.218659 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.218414    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f744f29b-e303-4c34-a559-f13c4593d522-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:20:12.218659 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.218425    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f744f29b-e303-4c34-a559-f13c4593d522-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:20:12.218659 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.218459    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") on node \"ip-10-0-131-33.ec2.internal\" "
Sep 19 09:20:12.250996 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.250973    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439") on node "ip-10-0-131-33.ec2.internal"
Sep 19 09:20:12.319526 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.319489    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-2d6dfd8a-b46b-40f2-b155-d1d451878968\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02179d6349ea07439\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"csi-c320ebc569d7b9ebdd42ef806151d9635f5043b75f1f3c3fb1ba2fa3869bb4bc\""
Sep 19 09:20:12.327717 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.327681    2577 generic.go:356] "Generic (PLEG): container finished" podID="f744f29b-e303-4c34-a559-f13c4593d522" containerID="8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13" exitCode=0
Sep 19 09:20:12.327919 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.327732    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"
Sep 19 09:20:12.327919 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.327763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" event={"ID":"f744f29b-e303-4c34-a559-f13c4593d522","Type":"ContainerDied","Data":"8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13"}
Sep 19 09:20:12.327919 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.327820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" event={"ID":"f744f29b-e303-4c34-a559-f13c4593d522","Type":"ContainerDied","Data":"9ef131dec727f66889e55bbc30d2370cbada6c1083e9814a53208b96b69d8615"}
Sep 19 09:20:12.327919 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.327844    2577 scope.go:117] "RemoveContainer" containerID="8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13"
Sep 19 09:20:12.338770 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.338748    2577 scope.go:117] "RemoveContainer" containerID="8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13"
Sep 19 09:20:12.339096 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:20:12.339073    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13\": container with ID starting with 8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13 not found: ID does not exist" containerID="8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13"
Sep 19 09:20:12.339142 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.339107    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13"} err="failed to get container status \"8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13\": rpc error: code = NotFound desc = could not find container \"8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13\": container with ID starting with 8219cb66269820f31db4590aebe197501a9d7c0a3dcc655dbd26c01166b4fa13 not found: ID does not exist"
Sep 19 09:20:12.351737 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.351707    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"]
Sep 19 09:20:12.356940 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.356907    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf"]
Sep 19 09:20:12.510222 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.510140    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f744f29b-e303-4c34-a559-f13c4593d522" path="/var/lib/kubelet/pods/f744f29b-e303-4c34-a559-f13c4593d522/volumes"
Sep 19 09:20:12.998183 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:12.998134    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-2dhnf" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 19 09:20:21.307902 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:21.307810    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"]
Sep 19 09:20:21.312698 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:21.312669    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8hw5p"]
Sep 19 09:20:22.503810 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:22.503770    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a266eaa1-09dd-4b72-8726-27225469cbb0" path="/var/lib/kubelet/pods/a266eaa1-09dd-4b72-8726-27225469cbb0/volumes"
Sep 19 09:20:31.787054 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.787015    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"]
Sep 19 09:20:31.787699 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.787505    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main"
Sep 19 09:20:31.787699 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.787522    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main"
Sep 19 09:20:31.787699 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.787532    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="storage-initializer"
Sep 19 09:20:31.787699 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.787539    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="storage-initializer"
Sep 19 09:20:31.787699 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.787551    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main"
Sep 19 09:20:31.787699 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.787556    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main"
Sep 19 09:20:31.787699 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.787623    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="693cddac-c2d7-47ec-a062-c563f431a11b" containerName="main"
Sep 19 09:20:31.787699 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.787633    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f744f29b-e303-4c34-a559-f13c4593d522" containerName="main"
Sep 19 09:20:31.790701 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.790680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.793407 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.793381    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 19 09:20:31.793523 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.793460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 19 09:20:31.793584 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.793460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-pwr2h\""
Sep 19 09:20:31.793642 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.793460    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:20:31.793816 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.793801    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 19 09:20:31.793885 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.793831    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 19 09:20:31.803269 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.803244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"]
Sep 19 09:20:31.887605 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.887576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-model-cache\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.887759 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.887610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-dshm\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.887759 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.887630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-home\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.887759 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.887654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kserve-provision-location\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.887759 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.887687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t85dj\" (UniqueName: \"kubernetes.io/projected/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kube-api-access-t85dj\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.887759 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.887706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tls-certs\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.887759 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.887735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tmp-dir\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.887960 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.887808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-cabundle-cert\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.988319 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.988285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-model-cache\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.988506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.988325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-dshm\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.988506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.988349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-home\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.988506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.988397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kserve-provision-location\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.988506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.988427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t85dj\" (UniqueName: \"kubernetes.io/projected/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kube-api-access-t85dj\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.988506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.988458    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tls-certs\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.988506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.988493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tmp-dir\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.988815 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.988571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-cabundle-cert\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.988815 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.988757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-model-cache\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.989099 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.989068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kserve-provision-location\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.989311 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.989288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-home\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.989461 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.989340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-cabundle-cert\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.989461 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.989340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tmp-dir\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.991035 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.991012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-dshm\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:31.991491 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:31.991462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tls-certs\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:32.000757 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:32.000736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t85dj\" (UniqueName: \"kubernetes.io/projected/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kube-api-access-t85dj\") pod \"tls-verification-test-kserve-65bc48d74c-pz7nq\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:32.102878 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:32.102831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:32.381937 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:32.381905    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"]
Sep 19 09:20:32.384260 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:20:32.384232    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e0bdbc_ec40_48d9_821c_b4cf77ba4884.slice/crio-da6bc7ff7237192dc30142b1ec185956d231289034e7d47a8d9271f7aa0ff444 WatchSource:0}: Error finding container da6bc7ff7237192dc30142b1ec185956d231289034e7d47a8d9271f7aa0ff444: Status 404 returned error can't find the container with id da6bc7ff7237192dc30142b1ec185956d231289034e7d47a8d9271f7aa0ff444
Sep 19 09:20:32.441427 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:32.441398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" event={"ID":"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884","Type":"ContainerStarted","Data":"da6bc7ff7237192dc30142b1ec185956d231289034e7d47a8d9271f7aa0ff444"}
Sep 19 09:20:33.448298 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:33.448252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" event={"ID":"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884","Type":"ContainerStarted","Data":"bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a"}
Sep 19 09:20:34.856019 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:34.855988    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:20:34.862200 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:34.862179    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:20:35.466021 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:35.465985    2577 generic.go:356] "Generic (PLEG): container finished" podID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerID="bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a" exitCode=0
Sep 19 09:20:35.466224 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:35.466065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" event={"ID":"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884","Type":"ContainerDied","Data":"bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a"}
Sep 19 09:20:36.473844 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:36.473807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" event={"ID":"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884","Type":"ContainerStarted","Data":"192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92"}
Sep 19 09:20:36.499762 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:36.499703    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podStartSLOduration=3.196230561 podStartE2EDuration="5.499688118s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.303457557s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:20:36.497142072 +0000 UTC m=+3604.178725506" watchObservedRunningTime="2026-09-19 09:20:36.499688118 +0000 UTC m=+3604.181271550"
Sep 19 09:20:37.103626 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:37.103583    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:42.103213 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:42.103168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:20:42.105111 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:42.105085    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:20:52.103652 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:20:52.103607    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:21:02.103871 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:02.103825    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:21:12.103290 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:12.103243    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:21:22.104091 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:22.104047    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:21:32.104038 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:32.103992    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:21:36.131936 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:36.131895    2577 scope.go:117] "RemoveContainer" containerID="fb7da6382b808d2fac3478bb1c97912d2c6fa83faae0dd83b0570763911dd5eb"
Sep 19 09:21:42.103939 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:42.103895    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:21:52.103644 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:52.103546    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:21:58.707348 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.707308    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 19 09:21:58.711675 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.711649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.715732 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.715693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 19 09:21:58.715877 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.715693    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-6qr9j\""
Sep 19 09:21:58.715877 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.715781    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:21:58.719915 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.719890    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 19 09:21:58.720262 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.720241    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 19 09:21:58.720453 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.720422    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 19 09:21:58.720553 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.720496    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-dm7px\""
Sep 19 09:21:58.730585 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.730556    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 19 09:21:58.759461 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.759426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.759620 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.759484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.759620 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.759524    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkwv\" (UniqueName: \"kubernetes.io/projected/17271926-d40e-4ed9-8ec5-c520844f5c1f-kube-api-access-7mkwv\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.759620 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.759598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.759734 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.759670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.759734 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.759719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17271926-d40e-4ed9-8ec5-c520844f5c1f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.759804 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.759745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.759804 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.759783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17271926-d40e-4ed9-8ec5-c520844f5c1f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.860424 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17271926-d40e-4ed9-8ec5-c520844f5c1f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.860624 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.860624 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.860624 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860548    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkwv\" (UniqueName: \"kubernetes.io/projected/17271926-d40e-4ed9-8ec5-c520844f5c1f-kube-api-access-7mkwv\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.860624 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.860870 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.860870 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17271926-d40e-4ed9-8ec5-c520844f5c1f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.860870 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.860870 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.861121 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.861121 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.860978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17271926-d40e-4ed9-8ec5-c520844f5c1f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.861304 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.861281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.861408 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.861325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.862936 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.862916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.863307 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.863281    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17271926-d40e-4ed9-8ec5-c520844f5c1f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:58.869674 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:58.869646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkwv\" (UniqueName: \"kubernetes.io/projected/17271926-d40e-4ed9-8ec5-c520844f5c1f-kube-api-access-7mkwv\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:59.024014 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:59.023924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:21:59.302556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:59.302504    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 19 09:21:59.303835 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:21:59.303803    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17271926_d40e_4ed9_8ec5_c520844f5c1f.slice/crio-f0916ea0dec8d8c1b3f9ef1c5000bfd0e55af0c0b84928e6770ca0a19f4e5faa WatchSource:0}: Error finding container f0916ea0dec8d8c1b3f9ef1c5000bfd0e55af0c0b84928e6770ca0a19f4e5faa: Status 404 returned error can't find the container with id f0916ea0dec8d8c1b3f9ef1c5000bfd0e55af0c0b84928e6770ca0a19f4e5faa
Sep 19 09:21:59.952014 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:59.951972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"17271926-d40e-4ed9-8ec5-c520844f5c1f","Type":"ContainerStarted","Data":"ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9"}
Sep 19 09:21:59.952440 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:21:59.952048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"17271926-d40e-4ed9-8ec5-c520844f5c1f","Type":"ContainerStarted","Data":"f0916ea0dec8d8c1b3f9ef1c5000bfd0e55af0c0b84928e6770ca0a19f4e5faa"}
Sep 19 09:22:01.965070 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:01.965029    2577 generic.go:356] "Generic (PLEG): container finished" podID="17271926-d40e-4ed9-8ec5-c520844f5c1f" containerID="ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9" exitCode=0
Sep 19 09:22:01.965480 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:01.965114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"17271926-d40e-4ed9-8ec5-c520844f5c1f","Type":"ContainerDied","Data":"ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9"}
Sep 19 09:22:02.103736 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:02.103694    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:22:02.973648 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:02.973612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"17271926-d40e-4ed9-8ec5-c520844f5c1f","Type":"ContainerStarted","Data":"9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53"}
Sep 19 09:22:02.996081 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:02.996016    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.643817531 podStartE2EDuration="4.995997255s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.352179724s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:22:02.992820848 +0000 UTC m=+3690.674404294" watchObservedRunningTime="2026-09-19 09:22:02.995997255 +0000 UTC m=+3690.677580712"
Sep 19 09:22:12.103955 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:12.103905    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:22:22.103425 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:22.103381    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:22:32.113723 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:32.113685    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:22:32.121975 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:32.121947    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:22:37.720434 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:37.720395    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"]
Sep 19 09:22:52.752454 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:52.752358    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" containerID="cri-o://192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92" gracePeriod=45
Sep 19 09:22:53.115064 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.115012    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": dial tcp 10.133.0.85:8000: connect: connection refused"
Sep 19 09:22:53.520864 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.520835    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:22:53.561875 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.561841    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tmp-dir\") pod \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") "
Sep 19 09:22:53.562051 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.561888    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-dshm\") pod \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") "
Sep 19 09:22:53.562051 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.561908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kserve-provision-location\") pod \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") "
Sep 19 09:22:53.562051 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.561929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-cabundle-cert\") pod \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") "
Sep 19 09:22:53.562051 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.562001    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tls-certs\") pod \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") "
Sep 19 09:22:53.562051 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.562022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-model-cache\") pod \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") "
Sep 19 09:22:53.562051 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.562044    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-home\") pod \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") "
Sep 19 09:22:53.562402 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.562081    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t85dj\" (UniqueName: \"kubernetes.io/projected/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kube-api-access-t85dj\") pod \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\" (UID: \"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884\") "
Sep 19 09:22:53.562653 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.562619    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" (UID: "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:22:53.562913 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.562711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" (UID: "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:22:53.563240 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.563186    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-home" (OuterVolumeSpecName: "home") pod "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" (UID: "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:22:53.565477 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.565451    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-dshm" (OuterVolumeSpecName: "dshm") pod "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" (UID: "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:22:53.565477 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.565469    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kube-api-access-t85dj" (OuterVolumeSpecName: "kube-api-access-t85dj") pod "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" (UID: "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884"). InnerVolumeSpecName "kube-api-access-t85dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:22:53.565644 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.565535    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" (UID: "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:22:53.581143 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.581108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" (UID: "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:22:53.589265 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.589236    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" (UID: "e6e0bdbc-ec40-48d9-821c-b4cf77ba4884"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:22:53.663311 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.663208    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:22:53.663311 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.663241    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:22:53.663311 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.663252    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:22:53.663311 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.663265    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t85dj\" (UniqueName: \"kubernetes.io/projected/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kube-api-access-t85dj\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:22:53.663311 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.663277    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:22:53.663311 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.663288    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:22:53.663311 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.663299    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:22:53.663311 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:53.663310    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:22:54.294182 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.294147    2577 generic.go:356] "Generic (PLEG): container finished" podID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerID="192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92" exitCode=0
Sep 19 09:22:54.294764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.294201    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"
Sep 19 09:22:54.294764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.294241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" event={"ID":"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884","Type":"ContainerDied","Data":"192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92"}
Sep 19 09:22:54.294764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.294310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq" event={"ID":"e6e0bdbc-ec40-48d9-821c-b4cf77ba4884","Type":"ContainerDied","Data":"da6bc7ff7237192dc30142b1ec185956d231289034e7d47a8d9271f7aa0ff444"}
Sep 19 09:22:54.294764 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.294348    2577 scope.go:117] "RemoveContainer" containerID="192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92"
Sep 19 09:22:54.304493 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.304468    2577 scope.go:117] "RemoveContainer" containerID="bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a"
Sep 19 09:22:54.313546 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.313517    2577 scope.go:117] "RemoveContainer" containerID="192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92"
Sep 19 09:22:54.313856 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:22:54.313835    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92\": container with ID starting with 192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92 not found: ID does not exist" containerID="192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92"
Sep 19 09:22:54.313914 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.313866    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92"} err="failed to get container status \"192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92\": rpc error: code = NotFound desc = could not find container \"192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92\": container with ID starting with 192ffc6662d9ae13b04338d7bd38ee7bd3968e3a6cdf710d1882f65ffd097d92 not found: ID does not exist"
Sep 19 09:22:54.313914 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.313886    2577 scope.go:117] "RemoveContainer" containerID="bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a"
Sep 19 09:22:54.314187 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:22:54.314170    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a\": container with ID starting with bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a not found: ID does not exist" containerID="bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a"
Sep 19 09:22:54.314238 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.314190    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a"} err="failed to get container status \"bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a\": rpc error: code = NotFound desc = could not find container \"bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a\": container with ID starting with bdd9bf99e912f8ec8c60a144cdcf244ca878a54c1e00fba81cc7f359a601c86a not found: ID does not exist"
Sep 19 09:22:54.319483 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.319456    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"]
Sep 19 09:22:54.323212 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.323183    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-pz7nq"]
Sep 19 09:22:54.504821 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:22:54.504789    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" path="/var/lib/kubelet/pods/e6e0bdbc-ec40-48d9-821c-b4cf77ba4884/volumes"
Sep 19 09:23:34.069714 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.069672    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"]
Sep 19 09:23:34.070276 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.070262    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main"
Sep 19 09:23:34.070355 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.070283    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main"
Sep 19 09:23:34.070355 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.070322    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="storage-initializer"
Sep 19 09:23:34.070355 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.070332    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="storage-initializer"
Sep 19 09:23:34.070544 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.070469    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6e0bdbc-ec40-48d9-821c-b4cf77ba4884" containerName="main"
Sep 19 09:23:34.075460 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.075437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.078324 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.078298    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:23:34.078486 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.078318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 19 09:23:34.078486 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.078317    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 19 09:23:34.078486 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.078426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 19 09:23:34.078486 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.078433    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 19 09:23:34.078486 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.078476    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-tjwkk\""
Sep 19 09:23:34.086190 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.086169    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"]
Sep 19 09:23:34.128759 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.128728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d4eea5-f139-4cd6-9dcc-94a46785c492-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.128941 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.128764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsd6\" (UniqueName: \"kubernetes.io/projected/01d4eea5-f139-4cd6-9dcc-94a46785c492-kube-api-access-mhsd6\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.128941 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.128789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.128941 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.128890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.129092 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.128944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-home\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.129092 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.128991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.129092 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.129037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-dshm\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.129092 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.129055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d4eea5-f139-4cd6-9dcc-94a46785c492-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230262 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-home\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-dshm\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d4eea5-f139-4cd6-9dcc-94a46785c492-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d4eea5-f139-4cd6-9dcc-94a46785c492-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230757 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsd6\" (UniqueName: \"kubernetes.io/projected/01d4eea5-f139-4cd6-9dcc-94a46785c492-kube-api-access-mhsd6\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230757 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230757 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230899 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-home\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230942 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.230996 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.230970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.231213 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.231194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d4eea5-f139-4cd6-9dcc-94a46785c492-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.232972 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.232948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-dshm\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.233185 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.233170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d4eea5-f139-4cd6-9dcc-94a46785c492-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.239687 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.239661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsd6\" (UniqueName: \"kubernetes.io/projected/01d4eea5-f139-4cd6-9dcc-94a46785c492-kube-api-access-mhsd6\") pod \"prestop-hook-test-kserve-d957cb699-wqz64\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.389384 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.389335    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-tjwkk\""
Sep 19 09:23:34.397446 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.397420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:34.687001 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.686971    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"]
Sep 19 09:23:34.688796 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:23:34.688752    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d4eea5_f139_4cd6_9dcc_94a46785c492.slice/crio-1fd3cec5e73ec97e6db464493daad9870f13ff27c241555e5e3213ebab75ce36 WatchSource:0}: Error finding container 1fd3cec5e73ec97e6db464493daad9870f13ff27c241555e5e3213ebab75ce36: Status 404 returned error can't find the container with id 1fd3cec5e73ec97e6db464493daad9870f13ff27c241555e5e3213ebab75ce36
Sep 19 09:23:34.693469 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:34.693443    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 19 09:23:35.544840 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:35.544805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" event={"ID":"01d4eea5-f139-4cd6-9dcc-94a46785c492","Type":"ContainerStarted","Data":"a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338"}
Sep 19 09:23:35.545262 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:35.544858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" event={"ID":"01d4eea5-f139-4cd6-9dcc-94a46785c492","Type":"ContainerStarted","Data":"1fd3cec5e73ec97e6db464493daad9870f13ff27c241555e5e3213ebab75ce36"}
Sep 19 09:23:36.163011 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:36.162978    2577 scope.go:117] "RemoveContainer" containerID="d2eae3bb330871891de919fa977114eb05f3e33e1abaee36d906789f6d1ae4b4"
Sep 19 09:23:37.559107 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:37.559074    2577 generic.go:356] "Generic (PLEG): container finished" podID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerID="a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338" exitCode=0
Sep 19 09:23:37.559549 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:37.559145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" event={"ID":"01d4eea5-f139-4cd6-9dcc-94a46785c492","Type":"ContainerDied","Data":"a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338"}
Sep 19 09:23:38.568072 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:38.568023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" event={"ID":"01d4eea5-f139-4cd6-9dcc-94a46785c492","Type":"ContainerStarted","Data":"a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480"}
Sep 19 09:23:38.594023 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:38.593957    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podStartSLOduration=2.298520117 podStartE2EDuration="4.593941966s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.295421849s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:23:38.590985915 +0000 UTC m=+3786.272569350" watchObservedRunningTime="2026-09-19 09:23:38.593941966 +0000 UTC m=+3786.275525398"
Sep 19 09:23:39.397600 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:39.397547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:44.397884 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:44.397847    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:23:44.399927 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:44.399900    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:23:54.398676 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:23:54.398627    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:24:01.986285 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:01.986233    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 19 09:24:04.398274 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:04.398223    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:24:14.398185 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:14.398139    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:24:17.009184 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.009136    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="17271926-d40e-4ed9-8ec5-c520844f5c1f" containerName="main" containerID="cri-o://9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53" gracePeriod=45
Sep 19 09:24:17.756482 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.756451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:24:17.813556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.813447    2577 generic.go:356] "Generic (PLEG): container finished" podID="17271926-d40e-4ed9-8ec5-c520844f5c1f" containerID="9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53" exitCode=0
Sep 19 09:24:17.813556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.813494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"17271926-d40e-4ed9-8ec5-c520844f5c1f","Type":"ContainerDied","Data":"9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53"}
Sep 19 09:24:17.813556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.813534    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 19 09:24:17.813854 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.813595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"17271926-d40e-4ed9-8ec5-c520844f5c1f","Type":"ContainerDied","Data":"f0916ea0dec8d8c1b3f9ef1c5000bfd0e55af0c0b84928e6770ca0a19f4e5faa"}
Sep 19 09:24:17.813854 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.813629    2577 scope.go:117] "RemoveContainer" containerID="9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53"
Sep 19 09:24:17.819897 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.819876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-dshm\") pod \"17271926-d40e-4ed9-8ec5-c520844f5c1f\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") "
Sep 19 09:24:17.820004 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.819926    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-model-cache\") pod \"17271926-d40e-4ed9-8ec5-c520844f5c1f\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") "
Sep 19 09:24:17.820004 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.819948    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-home\") pod \"17271926-d40e-4ed9-8ec5-c520844f5c1f\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") "
Sep 19 09:24:17.820004 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.819971    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mkwv\" (UniqueName: \"kubernetes.io/projected/17271926-d40e-4ed9-8ec5-c520844f5c1f-kube-api-access-7mkwv\") pod \"17271926-d40e-4ed9-8ec5-c520844f5c1f\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") "
Sep 19 09:24:17.820004 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.819986    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17271926-d40e-4ed9-8ec5-c520844f5c1f-cabundle-cert\") pod \"17271926-d40e-4ed9-8ec5-c520844f5c1f\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") "
Sep 19 09:24:17.820225 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.820029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-tmp-dir\") pod \"17271926-d40e-4ed9-8ec5-c520844f5c1f\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") "
Sep 19 09:24:17.820225 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.820046    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-kserve-provision-location\") pod \"17271926-d40e-4ed9-8ec5-c520844f5c1f\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") "
Sep 19 09:24:17.820225 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.820069    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17271926-d40e-4ed9-8ec5-c520844f5c1f-tls-certs\") pod \"17271926-d40e-4ed9-8ec5-c520844f5c1f\" (UID: \"17271926-d40e-4ed9-8ec5-c520844f5c1f\") "
Sep 19 09:24:17.820409 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.820238    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "17271926-d40e-4ed9-8ec5-c520844f5c1f" (UID: "17271926-d40e-4ed9-8ec5-c520844f5c1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:17.820409 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.820331    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:24:17.820923 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.820900    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-home" (OuterVolumeSpecName: "home") pod "17271926-d40e-4ed9-8ec5-c520844f5c1f" (UID: "17271926-d40e-4ed9-8ec5-c520844f5c1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:17.821205 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.821180    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17271926-d40e-4ed9-8ec5-c520844f5c1f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17271926-d40e-4ed9-8ec5-c520844f5c1f" (UID: "17271926-d40e-4ed9-8ec5-c520844f5c1f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:24:17.822882 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.822857    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17271926-d40e-4ed9-8ec5-c520844f5c1f-kube-api-access-7mkwv" (OuterVolumeSpecName: "kube-api-access-7mkwv") pod "17271926-d40e-4ed9-8ec5-c520844f5c1f" (UID: "17271926-d40e-4ed9-8ec5-c520844f5c1f"). InnerVolumeSpecName "kube-api-access-7mkwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:24:17.823272 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.823250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-dshm" (OuterVolumeSpecName: "dshm") pod "17271926-d40e-4ed9-8ec5-c520844f5c1f" (UID: "17271926-d40e-4ed9-8ec5-c520844f5c1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:17.823466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.823440    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17271926-d40e-4ed9-8ec5-c520844f5c1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17271926-d40e-4ed9-8ec5-c520844f5c1f" (UID: "17271926-d40e-4ed9-8ec5-c520844f5c1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:24:17.827143 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.827122    2577 scope.go:117] "RemoveContainer" containerID="ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9"
Sep 19 09:24:17.837324 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.837283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17271926-d40e-4ed9-8ec5-c520844f5c1f" (UID: "17271926-d40e-4ed9-8ec5-c520844f5c1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:17.842925 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.842899    2577 scope.go:117] "RemoveContainer" containerID="9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53"
Sep 19 09:24:17.843249 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:24:17.843228    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53\": container with ID starting with 9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53 not found: ID does not exist" containerID="9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53"
Sep 19 09:24:17.843334 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.843258    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53"} err="failed to get container status \"9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53\": rpc error: code = NotFound desc = could not find container \"9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53\": container with ID starting with 9b1bcd61b6716419c52a92d9bbe297d760e022b57e14094aa3a17a451234cd53 not found: ID does not exist"
Sep 19 09:24:17.843334 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.843278    2577 scope.go:117] "RemoveContainer" containerID="ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9"
Sep 19 09:24:17.843620 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:24:17.843603    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9\": container with ID starting with ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9 not found: ID does not exist" containerID="ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9"
Sep 19 09:24:17.843671 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.843625    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9"} err="failed to get container status \"ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9\": rpc error: code = NotFound desc = could not find container \"ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9\": container with ID starting with ed8147cfeb180867f3a2718ca610f07f239b8465ef1e733d15c744f9140010a9 not found: ID does not exist"
Sep 19 09:24:17.843777 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.843756    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17271926-d40e-4ed9-8ec5-c520844f5c1f" (UID: "17271926-d40e-4ed9-8ec5-c520844f5c1f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:17.921215 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.921177    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:24:17.921215 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.921210    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mkwv\" (UniqueName: \"kubernetes.io/projected/17271926-d40e-4ed9-8ec5-c520844f5c1f-kube-api-access-7mkwv\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:24:17.921215 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.921222    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17271926-d40e-4ed9-8ec5-c520844f5c1f-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:24:17.921466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.921231    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:24:17.921466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.921240    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:24:17.921466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.921248    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17271926-d40e-4ed9-8ec5-c520844f5c1f-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:24:17.921466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:17.921256    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17271926-d40e-4ed9-8ec5-c520844f5c1f-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:24:18.137731 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:18.137700    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 19 09:24:18.143376 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:18.143338    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 19 09:24:18.504567 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:18.504485    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17271926-d40e-4ed9-8ec5-c520844f5c1f" path="/var/lib/kubelet/pods/17271926-d40e-4ed9-8ec5-c520844f5c1f/volumes"
Sep 19 09:24:24.397864 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:24.397826    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:24:34.398618 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:34.398568    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:24:44.398290 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:44.398235    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:24:54.398465 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:24:54.398351    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:25:04.398045 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:04.397999    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:25:14.398599 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:14.398532    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:25:24.398357 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:24.398320    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Sep 19 09:25:34.408904 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:34.408864    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:25:34.421237 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:34.421204    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:25:34.891964 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:34.891934    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:25:34.901181 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:34.901157    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:25:38.531488 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.531452    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"]
Sep 19 09:25:38.556758 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.556715    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"]
Sep 19 09:25:38.557136 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.557119    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17271926-d40e-4ed9-8ec5-c520844f5c1f" containerName="storage-initializer"
Sep 19 09:25:38.557136 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.557140    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17271926-d40e-4ed9-8ec5-c520844f5c1f" containerName="storage-initializer"
Sep 19 09:25:38.557242 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.557166    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17271926-d40e-4ed9-8ec5-c520844f5c1f" containerName="main"
Sep 19 09:25:38.557242 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.557172    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="17271926-d40e-4ed9-8ec5-c520844f5c1f" containerName="main"
Sep 19 09:25:38.557242 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.557237    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="17271926-d40e-4ed9-8ec5-c520844f5c1f" containerName="main"
Sep 19 09:25:38.562207 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.562178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.576323 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.576285    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"]
Sep 19 09:25:38.649927 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.649894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4acc830-3cd0-49c9-9319-6b5101cb1700-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.650115 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.649961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-home\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.650115 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.650033    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.650115 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.650086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4acc830-3cd0-49c9-9319-6b5101cb1700-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.650115 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.650112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.650253 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.650141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-dshm\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.650253 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.650156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.650253 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.650176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-624cp\" (UniqueName: \"kubernetes.io/projected/d4acc830-3cd0-49c9-9319-6b5101cb1700-kube-api-access-624cp\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751059 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4acc830-3cd0-49c9-9319-6b5101cb1700-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751247 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751247 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-dshm\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751247 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751418 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-624cp\" (UniqueName: \"kubernetes.io/projected/d4acc830-3cd0-49c9-9319-6b5101cb1700-kube-api-access-624cp\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751418 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4acc830-3cd0-49c9-9319-6b5101cb1700-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751526 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-home\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751526 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751628 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751545    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751685 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751849 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.751952 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-home\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.752007 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.751947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4acc830-3cd0-49c9-9319-6b5101cb1700-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.753612 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.753589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-dshm\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.754094 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.754073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4acc830-3cd0-49c9-9319-6b5101cb1700-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.762156 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.762136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-624cp\" (UniqueName: \"kubernetes.io/projected/d4acc830-3cd0-49c9-9319-6b5101cb1700-kube-api-access-624cp\") pod \"prestop-hook-test-kserve-d957cb699-n7r4t\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:38.872810 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:38.872763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:39.154526 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:39.154486    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"]
Sep 19 09:25:39.154941 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:25:39.154913    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4acc830_3cd0_49c9_9319_6b5101cb1700.slice/crio-200f92c50ed66f6399782895228de261cc125c1bfb700a18391ea160c7d2ffdd WatchSource:0}: Error finding container 200f92c50ed66f6399782895228de261cc125c1bfb700a18391ea160c7d2ffdd: Status 404 returned error can't find the container with id 200f92c50ed66f6399782895228de261cc125c1bfb700a18391ea160c7d2ffdd
Sep 19 09:25:39.286183 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:39.286140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" event={"ID":"d4acc830-3cd0-49c9-9319-6b5101cb1700","Type":"ContainerStarted","Data":"82f0d33ca0faf303eaaa0af004acbfa799cc72d1d517ac5e47fe47c766d24839"}
Sep 19 09:25:39.286376 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:39.286202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" event={"ID":"d4acc830-3cd0-49c9-9319-6b5101cb1700","Type":"ContainerStarted","Data":"200f92c50ed66f6399782895228de261cc125c1bfb700a18391ea160c7d2ffdd"}
Sep 19 09:25:42.304458 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:42.304424    2577 generic.go:356] "Generic (PLEG): container finished" podID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerID="82f0d33ca0faf303eaaa0af004acbfa799cc72d1d517ac5e47fe47c766d24839" exitCode=0
Sep 19 09:25:42.304863 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:42.304495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" event={"ID":"d4acc830-3cd0-49c9-9319-6b5101cb1700","Type":"ContainerDied","Data":"82f0d33ca0faf303eaaa0af004acbfa799cc72d1d517ac5e47fe47c766d24839"}
Sep 19 09:25:43.313693 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:43.313650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" event={"ID":"d4acc830-3cd0-49c9-9319-6b5101cb1700","Type":"ContainerStarted","Data":"89410242d5dfbf51cab0a431be0562956369d40259d68ba9fec2e66929c2b9ee"}
Sep 19 09:25:43.337701 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:43.337646    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" podStartSLOduration=3.057201947 podStartE2EDuration="5.337628147s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.2804262s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:25:43.335563857 +0000 UTC m=+3911.017147314" watchObservedRunningTime="2026-09-19 09:25:43.337628147 +0000 UTC m=+3911.019211579"
Sep 19 09:25:43.873711 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:43.873664    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:48.873909 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:48.873856    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:25:48.875652 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:48.875617    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" podUID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 19 09:25:53.554911 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:53.554840    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main" containerID="cri-o://a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480" gracePeriod=45
Sep 19 09:25:54.313327 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.313296    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:25:54.386686 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.386643    2577 generic.go:356] "Generic (PLEG): container finished" podID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerID="a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480" exitCode=0
Sep 19 09:25:54.386872 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.386715    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"
Sep 19 09:25:54.386872 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.386716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" event={"ID":"01d4eea5-f139-4cd6-9dcc-94a46785c492","Type":"ContainerDied","Data":"a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480"}
Sep 19 09:25:54.386872 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.386843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64" event={"ID":"01d4eea5-f139-4cd6-9dcc-94a46785c492","Type":"ContainerDied","Data":"1fd3cec5e73ec97e6db464493daad9870f13ff27c241555e5e3213ebab75ce36"}
Sep 19 09:25:54.386872 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.386866    2577 scope.go:117] "RemoveContainer" containerID="a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480"
Sep 19 09:25:54.396861 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.396832    2577 scope.go:117] "RemoveContainer" containerID="a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338"
Sep 19 09:25:54.397049 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397023    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d4eea5-f139-4cd6-9dcc-94a46785c492-cabundle-cert\") pod \"01d4eea5-f139-4cd6-9dcc-94a46785c492\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") "
Sep 19 09:25:54.397162 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-model-cache\") pod \"01d4eea5-f139-4cd6-9dcc-94a46785c492\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") "
Sep 19 09:25:54.397162 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397108    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhsd6\" (UniqueName: \"kubernetes.io/projected/01d4eea5-f139-4cd6-9dcc-94a46785c492-kube-api-access-mhsd6\") pod \"01d4eea5-f139-4cd6-9dcc-94a46785c492\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") "
Sep 19 09:25:54.397303 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-home\") pod \"01d4eea5-f139-4cd6-9dcc-94a46785c492\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") "
Sep 19 09:25:54.397303 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397210    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d4eea5-f139-4cd6-9dcc-94a46785c492-tls-certs\") pod \"01d4eea5-f139-4cd6-9dcc-94a46785c492\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") "
Sep 19 09:25:54.397303 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397233    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-kserve-provision-location\") pod \"01d4eea5-f139-4cd6-9dcc-94a46785c492\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") "
Sep 19 09:25:54.397303 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397279    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-tmp-dir\") pod \"01d4eea5-f139-4cd6-9dcc-94a46785c492\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") "
Sep 19 09:25:54.397532 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-dshm\") pod \"01d4eea5-f139-4cd6-9dcc-94a46785c492\" (UID: \"01d4eea5-f139-4cd6-9dcc-94a46785c492\") "
Sep 19 09:25:54.397532 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397502    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d4eea5-f139-4cd6-9dcc-94a46785c492-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01d4eea5-f139-4cd6-9dcc-94a46785c492" (UID: "01d4eea5-f139-4cd6-9dcc-94a46785c492"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:25:54.397673 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.397657    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d4eea5-f139-4cd6-9dcc-94a46785c492-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:25:54.398744 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.398488    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-home" (OuterVolumeSpecName: "home") pod "01d4eea5-f139-4cd6-9dcc-94a46785c492" (UID: "01d4eea5-f139-4cd6-9dcc-94a46785c492"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:25:54.398960 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.398922    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-model-cache" (OuterVolumeSpecName: "model-cache") pod "01d4eea5-f139-4cd6-9dcc-94a46785c492" (UID: "01d4eea5-f139-4cd6-9dcc-94a46785c492"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:25:54.401439 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.401411    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d4eea5-f139-4cd6-9dcc-94a46785c492-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01d4eea5-f139-4cd6-9dcc-94a46785c492" (UID: "01d4eea5-f139-4cd6-9dcc-94a46785c492"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:25:54.401543 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.401433    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d4eea5-f139-4cd6-9dcc-94a46785c492-kube-api-access-mhsd6" (OuterVolumeSpecName: "kube-api-access-mhsd6") pod "01d4eea5-f139-4cd6-9dcc-94a46785c492" (UID: "01d4eea5-f139-4cd6-9dcc-94a46785c492"). InnerVolumeSpecName "kube-api-access-mhsd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:25:54.401543 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.401504    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-dshm" (OuterVolumeSpecName: "dshm") pod "01d4eea5-f139-4cd6-9dcc-94a46785c492" (UID: "01d4eea5-f139-4cd6-9dcc-94a46785c492"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:25:54.412254 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.412211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01d4eea5-f139-4cd6-9dcc-94a46785c492" (UID: "01d4eea5-f139-4cd6-9dcc-94a46785c492"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:25:54.420143 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.420111    2577 scope.go:117] "RemoveContainer" containerID="a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480"
Sep 19 09:25:54.420518 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:25:54.420491    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480\": container with ID starting with a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480 not found: ID does not exist" containerID="a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480"
Sep 19 09:25:54.420632 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.420527    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480"} err="failed to get container status \"a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480\": rpc error: code = NotFound desc = could not find container \"a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480\": container with ID starting with a875dfce181937464375fe13032b76707c771b268787de593bc809133bdfe480 not found: ID does not exist"
Sep 19 09:25:54.420632 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.420547    2577 scope.go:117] "RemoveContainer" containerID="a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338"
Sep 19 09:25:54.420868 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:25:54.420848    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338\": container with ID starting with a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338 not found: ID does not exist" containerID="a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338"
Sep 19 09:25:54.420934 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.420872    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338"} err="failed to get container status \"a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338\": rpc error: code = NotFound desc = could not find container \"a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338\": container with ID starting with a5300c9d73bf86f48ade320ff8fd229a9cb0887b31891f12299b16ee0272c338 not found: ID does not exist"
Sep 19 09:25:54.426139 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.426097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01d4eea5-f139-4cd6-9dcc-94a46785c492" (UID: "01d4eea5-f139-4cd6-9dcc-94a46785c492"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:25:54.498047 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.498012    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:25:54.498047 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.498041    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:25:54.498047 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.498053    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhsd6\" (UniqueName: \"kubernetes.io/projected/01d4eea5-f139-4cd6-9dcc-94a46785c492-kube-api-access-mhsd6\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:25:54.498291 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.498061    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:25:54.498291 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.498071    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d4eea5-f139-4cd6-9dcc-94a46785c492-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:25:54.498291 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.498084    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:25:54.498291 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.498094    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d4eea5-f139-4cd6-9dcc-94a46785c492-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:25:54.705115 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.705031    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"]
Sep 19 09:25:54.709957 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:54.709921    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-wqz64"]
Sep 19 09:25:56.504109 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:56.504070    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" path="/var/lib/kubelet/pods/01d4eea5-f139-4cd6-9dcc-94a46785c492/volumes"
Sep 19 09:25:58.873150 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:25:58.873106    2577 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" podUID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": dial tcp 10.133.0.88:8000: connect: connection refused"
Sep 19 09:26:00.109408 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:26:00.109339    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"]
Sep 19 09:26:15.132986 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:26:15.132915    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" podUID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerName="main" containerID="cri-o://89410242d5dfbf51cab0a431be0562956369d40259d68ba9fec2e66929c2b9ee" gracePeriod=45
Sep 19 09:27:00.760100 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.760071    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-d957cb699-n7r4t_d4acc830-3cd0-49c9-9319-6b5101cb1700/main/0.log"
Sep 19 09:27:00.760605 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.760501    2577 generic.go:356] "Generic (PLEG): container finished" podID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerID="89410242d5dfbf51cab0a431be0562956369d40259d68ba9fec2e66929c2b9ee" exitCode=137
Sep 19 09:27:00.760605 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.760569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" event={"ID":"d4acc830-3cd0-49c9-9319-6b5101cb1700","Type":"ContainerDied","Data":"89410242d5dfbf51cab0a431be0562956369d40259d68ba9fec2e66929c2b9ee"}
Sep 19 09:27:00.788885 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.788860    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-d957cb699-n7r4t_d4acc830-3cd0-49c9-9319-6b5101cb1700/main/0.log"
Sep 19 09:27:00.791510 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.791488    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:27:00.902805 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.902715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-kserve-provision-location\") pod \"d4acc830-3cd0-49c9-9319-6b5101cb1700\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") "
Sep 19 09:27:00.902805 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.902777    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-tmp-dir\") pod \"d4acc830-3cd0-49c9-9319-6b5101cb1700\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") "
Sep 19 09:27:00.903013 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.902814    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-model-cache\") pod \"d4acc830-3cd0-49c9-9319-6b5101cb1700\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") "
Sep 19 09:27:00.903013 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.902839    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4acc830-3cd0-49c9-9319-6b5101cb1700-cabundle-cert\") pod \"d4acc830-3cd0-49c9-9319-6b5101cb1700\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") "
Sep 19 09:27:00.903013 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.902870    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-624cp\" (UniqueName: \"kubernetes.io/projected/d4acc830-3cd0-49c9-9319-6b5101cb1700-kube-api-access-624cp\") pod \"d4acc830-3cd0-49c9-9319-6b5101cb1700\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") "
Sep 19 09:27:00.903176 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-home\") pod \"d4acc830-3cd0-49c9-9319-6b5101cb1700\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") "
Sep 19 09:27:00.903176 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903094    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4acc830-3cd0-49c9-9319-6b5101cb1700" (UID: "d4acc830-3cd0-49c9-9319-6b5101cb1700"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:00.903176 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903109    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4acc830-3cd0-49c9-9319-6b5101cb1700-tls-certs\") pod \"d4acc830-3cd0-49c9-9319-6b5101cb1700\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") "
Sep 19 09:27:00.903176 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903141    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4acc830-3cd0-49c9-9319-6b5101cb1700" (UID: "d4acc830-3cd0-49c9-9319-6b5101cb1700"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:00.903176 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-dshm\") pod \"d4acc830-3cd0-49c9-9319-6b5101cb1700\" (UID: \"d4acc830-3cd0-49c9-9319-6b5101cb1700\") "
Sep 19 09:27:00.903444 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903343    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4acc830-3cd0-49c9-9319-6b5101cb1700-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4acc830-3cd0-49c9-9319-6b5101cb1700" (UID: "d4acc830-3cd0-49c9-9319-6b5101cb1700"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:27:00.903444 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903418    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-home" (OuterVolumeSpecName: "home") pod "d4acc830-3cd0-49c9-9319-6b5101cb1700" (UID: "d4acc830-3cd0-49c9-9319-6b5101cb1700"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:00.903556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903507    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:00.903556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903525    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:00.903556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903540    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4acc830-3cd0-49c9-9319-6b5101cb1700-cabundle-cert\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:00.903556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.903553    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:00.905299 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.905269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4acc830-3cd0-49c9-9319-6b5101cb1700-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4acc830-3cd0-49c9-9319-6b5101cb1700" (UID: "d4acc830-3cd0-49c9-9319-6b5101cb1700"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:27:00.905539 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.905299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4acc830-3cd0-49c9-9319-6b5101cb1700-kube-api-access-624cp" (OuterVolumeSpecName: "kube-api-access-624cp") pod "d4acc830-3cd0-49c9-9319-6b5101cb1700" (UID: "d4acc830-3cd0-49c9-9319-6b5101cb1700"). InnerVolumeSpecName "kube-api-access-624cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:27:00.905539 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.905379    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-dshm" (OuterVolumeSpecName: "dshm") pod "d4acc830-3cd0-49c9-9319-6b5101cb1700" (UID: "d4acc830-3cd0-49c9-9319-6b5101cb1700"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:00.923182 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:00.923152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4acc830-3cd0-49c9-9319-6b5101cb1700" (UID: "d4acc830-3cd0-49c9-9319-6b5101cb1700"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:01.004410 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.004353    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4acc830-3cd0-49c9-9319-6b5101cb1700-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:01.004410 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.004404    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:01.004410 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.004414    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4acc830-3cd0-49c9-9319-6b5101cb1700-kserve-provision-location\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:01.004628 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.004423    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-624cp\" (UniqueName: \"kubernetes.io/projected/d4acc830-3cd0-49c9-9319-6b5101cb1700-kube-api-access-624cp\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:01.767245 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.767216    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-d957cb699-n7r4t_d4acc830-3cd0-49c9-9319-6b5101cb1700/main/0.log"
Sep 19 09:27:01.767703 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.767658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t" event={"ID":"d4acc830-3cd0-49c9-9319-6b5101cb1700","Type":"ContainerDied","Data":"200f92c50ed66f6399782895228de261cc125c1bfb700a18391ea160c7d2ffdd"}
Sep 19 09:27:01.767703 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.767681    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"
Sep 19 09:27:01.767793 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.767711    2577 scope.go:117] "RemoveContainer" containerID="89410242d5dfbf51cab0a431be0562956369d40259d68ba9fec2e66929c2b9ee"
Sep 19 09:27:01.777290 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.777272    2577 scope.go:117] "RemoveContainer" containerID="82f0d33ca0faf303eaaa0af004acbfa799cc72d1d517ac5e47fe47c766d24839"
Sep 19 09:27:01.791414 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.791386    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"]
Sep 19 09:27:01.794708 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:01.794683    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-n7r4t"]
Sep 19 09:27:02.504665 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:02.504629    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4acc830-3cd0-49c9-9319-6b5101cb1700" path="/var/lib/kubelet/pods/d4acc830-3cd0-49c9-9319-6b5101cb1700/volumes"
Sep 19 09:27:11.178277 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178234    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"]
Sep 19 09:27:11.178820 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178801    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main"
Sep 19 09:27:11.178876 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178824    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main"
Sep 19 09:27:11.178876 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178841    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerName="main"
Sep 19 09:27:11.178876 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178850    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerName="main"
Sep 19 09:27:11.178876 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178865    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerName="storage-initializer"
Sep 19 09:27:11.178876 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178875    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerName="storage-initializer"
Sep 19 09:27:11.179039 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178910    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="storage-initializer"
Sep 19 09:27:11.179039 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178919    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="storage-initializer"
Sep 19 09:27:11.179039 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.178994    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d4eea5-f139-4cd6-9dcc-94a46785c492" containerName="main"
Sep 19 09:27:11.179039 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.179006    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4acc830-3cd0-49c9-9319-6b5101cb1700" containerName="main"
Sep 19 09:27:11.183950 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.183925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.188432 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.188406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 19 09:27:11.188586 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.188413    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 19 09:27:11.188586 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.188418    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-zk87s\""
Sep 19 09:27:11.188742 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.188718    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 19 09:27:11.191986 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.191960    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"]
Sep 19 09:27:11.295074 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.295031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.295261 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.295082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.295261 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.295108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.295261 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.295168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.295261 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.295231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.295441 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.295261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx494\" (UniqueName: \"kubernetes.io/projected/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-kube-api-access-fx494\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.396400 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.396346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.396713 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.396685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.396863 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.396731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx494\" (UniqueName: \"kubernetes.io/projected/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-kube-api-access-fx494\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.396863 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.396777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.396863 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.396814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.396863 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.396842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.397089 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.396898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.397190 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.397164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.397436 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.397406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.399863 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.399840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.400119 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.400093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.409852 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.409816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx494\" (UniqueName: \"kubernetes.io/projected/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-kube-api-access-fx494\") pod \"rolling-upgrade-test-kserve-74b79d7f77-8qkmh\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.496035 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.495935    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:11.772939 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.772907    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"]
Sep 19 09:27:11.774616 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:27:11.774586    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcd2f75_97f3_4ebd_ba62_ef1d9d5eb941.slice/crio-a61aa98f37871fdbeee7d43b82cfba27a3b92846fc44ee0a7c8b5112b9066dec WatchSource:0}: Error finding container a61aa98f37871fdbeee7d43b82cfba27a3b92846fc44ee0a7c8b5112b9066dec: Status 404 returned error can't find the container with id a61aa98f37871fdbeee7d43b82cfba27a3b92846fc44ee0a7c8b5112b9066dec
Sep 19 09:27:11.824765 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:11.824726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh" event={"ID":"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941","Type":"ContainerStarted","Data":"a61aa98f37871fdbeee7d43b82cfba27a3b92846fc44ee0a7c8b5112b9066dec"}
Sep 19 09:27:12.836310 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:12.836268    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh" event={"ID":"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941","Type":"ContainerStarted","Data":"c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57"}
Sep 19 09:27:12.858525 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:12.858461    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh" podStartSLOduration=1.8584443560000001 podStartE2EDuration="1.858444356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:27:12.855836624 +0000 UTC m=+4000.537420059" watchObservedRunningTime="2026-09-19 09:27:12.858444356 +0000 UTC m=+4000.540027788"
Sep 19 09:27:13.496432 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:13.496384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:21.497074 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:21.497030    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:21.509969 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:21.509938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:21.895760 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:21.895730    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:54.882146 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:54.882061    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"]
Sep 19 09:27:54.892408 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:27:54.892342    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 09:27:54.892408 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 09:27:54.892408 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 09:27:54.892408 ip-10-0-131-33 kubenswrapper[2577]:  > containerID="c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57" cmd=["/bin/sleep","15"]
Sep 19 09:27:54.892688 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:54.892434    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh" message="" err=<
Sep 19 09:27:54.892688 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 09:27:54.892688 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 09:27:54.892688 ip-10-0-131-33 kubenswrapper[2577]:  >
Sep 19 09:27:54.892688 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:27:54.892451    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 09:27:54.892688 ip-10-0-131-33 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 09:27:54.892688 ip-10-0-131-33 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Sep 19 09:27:54.892688 ip-10-0-131-33 kubenswrapper[2577]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh" podUID="4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" containerName="main" containerID="cri-o://c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57"
Sep 19 09:27:54.892688 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:54.892506    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh" podUID="4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" containerName="main" containerID="cri-o://c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57" gracePeriod=60
Sep 19 09:27:55.045703 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:27:55.045667    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcd2f75_97f3_4ebd_ba62_ef1d9d5eb941.slice/crio-conmon-c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57.scope\": RecentStats: unable to find data in memory cache]"
Sep 19 09:27:55.253999 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.253969    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:55.389277 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.389238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx494\" (UniqueName: \"kubernetes.io/projected/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-kube-api-access-fx494\") pod \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") "
Sep 19 09:27:55.389481 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.389289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-home\") pod \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") "
Sep 19 09:27:55.389481 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.389345    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tls-certs\") pod \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") "
Sep 19 09:27:55.389481 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.389423    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-model-cache\") pod \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") "
Sep 19 09:27:55.389481 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.389447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tmp-dir\") pod \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") "
Sep 19 09:27:55.389481 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.389467    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-dshm\") pod \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\" (UID: \"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941\") "
Sep 19 09:27:55.389757 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.389677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-model-cache" (OuterVolumeSpecName: "model-cache") pod "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" (UID: "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:55.389757 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.389703    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" (UID: "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:55.389862 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.389774    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-home" (OuterVolumeSpecName: "home") pod "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" (UID: "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:55.391689 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.391664    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-kube-api-access-fx494" (OuterVolumeSpecName: "kube-api-access-fx494") pod "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" (UID: "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941"). InnerVolumeSpecName "kube-api-access-fx494". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:27:55.391689 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.391680    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-dshm" (OuterVolumeSpecName: "dshm") pod "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" (UID: "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:55.391858 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.391754    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" (UID: "4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:27:55.490157 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.490094    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tls-certs\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.490157 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.490115    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-model-cache\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.490157 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.490125    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-tmp-dir\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.490157 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.490133    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-dshm\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.490157 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.490141    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx494\" (UniqueName: \"kubernetes.io/projected/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-kube-api-access-fx494\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.490157 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:55.490149    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941-home\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:27:56.079354 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.079313    2577 generic.go:356] "Generic (PLEG): container finished" podID="4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" containerID="c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57" exitCode=0
Sep 19 09:27:56.079823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.079382    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"
Sep 19 09:27:56.079823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.079402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh" event={"ID":"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941","Type":"ContainerDied","Data":"c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57"}
Sep 19 09:27:56.079823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.079459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh" event={"ID":"4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941","Type":"ContainerDied","Data":"a61aa98f37871fdbeee7d43b82cfba27a3b92846fc44ee0a7c8b5112b9066dec"}
Sep 19 09:27:56.079823 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.079481    2577 scope.go:117] "RemoveContainer" containerID="c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57"
Sep 19 09:27:56.089188 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.089168    2577 scope.go:117] "RemoveContainer" containerID="c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57"
Sep 19 09:27:56.089483 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:27:56.089463    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57\": container with ID starting with c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57 not found: ID does not exist" containerID="c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57"
Sep 19 09:27:56.089536 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.089492    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57"} err="failed to get container status \"c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57\": rpc error: code = NotFound desc = could not find container \"c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57\": container with ID starting with c26b8d6d5cacc25704375cf50542ad3774cc9a6aa98782de0bf17839771abf57 not found: ID does not exist"
Sep 19 09:27:56.101241 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.101215    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"]
Sep 19 09:27:56.105612 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.105589    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-8qkmh"]
Sep 19 09:27:56.504597 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:27:56.504564    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" path="/var/lib/kubelet/pods/4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941/volumes"
Sep 19 09:28:44.031668 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.031628    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vwldn/must-gather-t64nw"]
Sep 19 09:28:44.032152 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.032004    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" containerName="main"
Sep 19 09:28:44.032152 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.032018    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" containerName="main"
Sep 19 09:28:44.032152 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.032093    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dcd2f75-97f3-4ebd-ba62-ef1d9d5eb941" containerName="main"
Sep 19 09:28:44.035249 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.035228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:28:44.038001 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.037975    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vwldn\"/\"default-dockercfg-9qrt6\""
Sep 19 09:28:44.038828 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.038804    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vwldn\"/\"openshift-service-ca.crt\""
Sep 19 09:28:44.038828 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.038810    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vwldn\"/\"kube-root-ca.crt\""
Sep 19 09:28:44.049539 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.049515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vwldn/must-gather-t64nw"]
Sep 19 09:28:44.093968 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.093945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwgz\" (UniqueName: \"kubernetes.io/projected/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-kube-api-access-fmwgz\") pod \"must-gather-t64nw\" (UID: \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\") " pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:28:44.094070 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.094000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-must-gather-output\") pod \"must-gather-t64nw\" (UID: \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\") " pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:28:44.195479 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.195441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-must-gather-output\") pod \"must-gather-t64nw\" (UID: \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\") " pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:28:44.195650 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.195508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwgz\" (UniqueName: \"kubernetes.io/projected/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-kube-api-access-fmwgz\") pod \"must-gather-t64nw\" (UID: \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\") " pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:28:44.195809 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.195788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-must-gather-output\") pod \"must-gather-t64nw\" (UID: \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\") " pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:28:44.203822 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.203790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwgz\" (UniqueName: \"kubernetes.io/projected/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-kube-api-access-fmwgz\") pod \"must-gather-t64nw\" (UID: \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\") " pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:28:44.344533 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.344442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:28:44.613440 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.613408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vwldn/must-gather-t64nw"]
Sep 19 09:28:44.615537 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:28:44.615510    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5765d31_65b8_4a7d_8132_cf9a36baa5ea.slice/crio-5dbacaface43638a7b8c33589726f8c606083fad85a7479f74542a84d2b37d6d WatchSource:0}: Error finding container 5dbacaface43638a7b8c33589726f8c606083fad85a7479f74542a84d2b37d6d: Status 404 returned error can't find the container with id 5dbacaface43638a7b8c33589726f8c606083fad85a7479f74542a84d2b37d6d
Sep 19 09:28:44.617288 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:44.617268    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 09:28:45.348199 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:45.348129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vwldn/must-gather-t64nw" event={"ID":"d5765d31-65b8-4a7d-8132-cf9a36baa5ea","Type":"ContainerStarted","Data":"5dbacaface43638a7b8c33589726f8c606083fad85a7479f74542a84d2b37d6d"}
Sep 19 09:28:51.387579 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:51.387529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vwldn/must-gather-t64nw" event={"ID":"d5765d31-65b8-4a7d-8132-cf9a36baa5ea","Type":"ContainerStarted","Data":"555098ca7cf6757601917ecffd38fa064b34e574839dcaf80bda18cb27acaaef"}
Sep 19 09:28:51.388182 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:51.388155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vwldn/must-gather-t64nw" event={"ID":"d5765d31-65b8-4a7d-8132-cf9a36baa5ea","Type":"ContainerStarted","Data":"3b03f8700c461c6f0cc540b0e71957b274f9b0f0930cdda5121b1a0e2d79b711"}
Sep 19 09:28:51.406562 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:28:51.406512    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vwldn/must-gather-t64nw" podStartSLOduration=1.433312438 podStartE2EDuration="7.406498836s" totalImagesPullingTime="5.973186398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:28:51.403718341 +0000 UTC m=+4099.085301773" watchObservedRunningTime="2026-09-19 09:28:51.406498836 +0000 UTC m=+4099.088082246"
Sep 19 09:29:06.513575 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:06.513540    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/0.log"
Sep 19 09:29:06.516790 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:06.516753    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/1.log"
Sep 19 09:29:06.529070 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:06.529050    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f9d67fdf8-x2dbm_ddfd0060-8f85-422c-85c1-9570edc61e0d/router/0.log"
Sep 19 09:29:07.265421 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:07.265358    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/0.log"
Sep 19 09:29:07.268651 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:07.268629    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/1.log"
Sep 19 09:29:07.285599 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:07.285571    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f9d67fdf8-x2dbm_ddfd0060-8f85-422c-85c1-9570edc61e0d/router/0.log"
Sep 19 09:29:08.261170 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:08.261139    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-9jnzl_b7f4c415-703f-4079-a006-5851c37d5491/manager/0.log"
Sep 19 09:29:08.380866 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:08.380830    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6zhpz_21f44ad0-93f6-45c6-8d2a-f36c85490def/limitador/0.log"
Sep 19 09:29:08.396024 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:08.395996    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-trd7d_71ac61a2-7f18-42cf-b280-e37a2f2e24e1/manager/0.log"
Sep 19 09:29:11.503348 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:11.503315    2577 generic.go:356] "Generic (PLEG): container finished" podID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" containerID="3b03f8700c461c6f0cc540b0e71957b274f9b0f0930cdda5121b1a0e2d79b711" exitCode=0
Sep 19 09:29:11.503830 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:11.503384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vwldn/must-gather-t64nw" event={"ID":"d5765d31-65b8-4a7d-8132-cf9a36baa5ea","Type":"ContainerDied","Data":"3b03f8700c461c6f0cc540b0e71957b274f9b0f0930cdda5121b1a0e2d79b711"}
Sep 19 09:29:11.503830 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:11.503688    2577 scope.go:117] "RemoveContainer" containerID="3b03f8700c461c6f0cc540b0e71957b274f9b0f0930cdda5121b1a0e2d79b711"
Sep 19 09:29:11.858675 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:11.858647    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vwldn_must-gather-t64nw_d5765d31-65b8-4a7d-8132-cf9a36baa5ea/gather/0.log"
Sep 19 09:29:12.408701 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.408659    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h99lz/must-gather-xj54d"]
Sep 19 09:29:12.413018 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.412995    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h99lz/must-gather-xj54d"
Sep 19 09:29:12.415980 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.415956    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h99lz\"/\"kube-root-ca.crt\""
Sep 19 09:29:12.416102 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.415985    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h99lz\"/\"openshift-service-ca.crt\""
Sep 19 09:29:12.416102 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.416050    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h99lz\"/\"default-dockercfg-zpdhd\""
Sep 19 09:29:12.424584 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.424558    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h99lz/must-gather-xj54d"]
Sep 19 09:29:12.538887 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.538855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29500085-8f83-4c17-9a50-ed29586d1360-must-gather-output\") pod \"must-gather-xj54d\" (UID: \"29500085-8f83-4c17-9a50-ed29586d1360\") " pod="openshift-must-gather-h99lz/must-gather-xj54d"
Sep 19 09:29:12.538887 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.538890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2krhv\" (UniqueName: \"kubernetes.io/projected/29500085-8f83-4c17-9a50-ed29586d1360-kube-api-access-2krhv\") pod \"must-gather-xj54d\" (UID: \"29500085-8f83-4c17-9a50-ed29586d1360\") " pod="openshift-must-gather-h99lz/must-gather-xj54d"
Sep 19 09:29:12.640063 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.640026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29500085-8f83-4c17-9a50-ed29586d1360-must-gather-output\") pod \"must-gather-xj54d\" (UID: \"29500085-8f83-4c17-9a50-ed29586d1360\") " pod="openshift-must-gather-h99lz/must-gather-xj54d"
Sep 19 09:29:12.640235 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.640168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2krhv\" (UniqueName: \"kubernetes.io/projected/29500085-8f83-4c17-9a50-ed29586d1360-kube-api-access-2krhv\") pod \"must-gather-xj54d\" (UID: \"29500085-8f83-4c17-9a50-ed29586d1360\") " pod="openshift-must-gather-h99lz/must-gather-xj54d"
Sep 19 09:29:12.640466 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.640449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/29500085-8f83-4c17-9a50-ed29586d1360-must-gather-output\") pod \"must-gather-xj54d\" (UID: \"29500085-8f83-4c17-9a50-ed29586d1360\") " pod="openshift-must-gather-h99lz/must-gather-xj54d"
Sep 19 09:29:12.649408 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.649380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2krhv\" (UniqueName: \"kubernetes.io/projected/29500085-8f83-4c17-9a50-ed29586d1360-kube-api-access-2krhv\") pod \"must-gather-xj54d\" (UID: \"29500085-8f83-4c17-9a50-ed29586d1360\") " pod="openshift-must-gather-h99lz/must-gather-xj54d"
Sep 19 09:29:12.722866 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.722787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h99lz/must-gather-xj54d"
Sep 19 09:29:12.996443 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:12.996408    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h99lz/must-gather-xj54d"]
Sep 19 09:29:12.997908 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:29:12.997879    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29500085_8f83_4c17_9a50_ed29586d1360.slice/crio-9ab89df606a3c14971174422966a788baf49019fe8a6ebad3d915d87d910b34b WatchSource:0}: Error finding container 9ab89df606a3c14971174422966a788baf49019fe8a6ebad3d915d87d910b34b: Status 404 returned error can't find the container with id 9ab89df606a3c14971174422966a788baf49019fe8a6ebad3d915d87d910b34b
Sep 19 09:29:13.513716 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:13.513679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h99lz/must-gather-xj54d" event={"ID":"29500085-8f83-4c17-9a50-ed29586d1360","Type":"ContainerStarted","Data":"9ab89df606a3c14971174422966a788baf49019fe8a6ebad3d915d87d910b34b"}
Sep 19 09:29:14.520464 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:14.520416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h99lz/must-gather-xj54d" event={"ID":"29500085-8f83-4c17-9a50-ed29586d1360","Type":"ContainerStarted","Data":"640557d8e9c581481b4f059a941a2c401ce70bc785722e5a13f5f8cb9d4c35d1"}
Sep 19 09:29:14.520867 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:14.520476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h99lz/must-gather-xj54d" event={"ID":"29500085-8f83-4c17-9a50-ed29586d1360","Type":"ContainerStarted","Data":"a5ce2842e62bd2a5b79b13712df7b9dc91f013cd5fe0a357730951901691e3a8"}
Sep 19 09:29:14.537707 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:14.537649    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h99lz/must-gather-xj54d" podStartSLOduration=1.8197814719999998 podStartE2EDuration="2.537629407s" totalImagesPullingTime="717.847935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:29:14.535671832 +0000 UTC m=+4122.217255277" watchObservedRunningTime="2026-09-19 09:29:14.537629407 +0000 UTC m=+4122.219212840"
Sep 19 09:29:15.195915 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:15.195862    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gqvcc_95d065d2-7e01-4e45-83a6-22e2646d868f/global-pull-secret-syncer/0.log"
Sep 19 09:29:15.391921 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:15.391890    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vs4dq_d385eb5e-e888-4a6a-9aee-5b23faffb0ec/konnectivity-agent/0.log"
Sep 19 09:29:15.414271 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:15.414240    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-33.ec2.internal_9fb7efdd43139469eaac4cb3e953207b/haproxy/0.log"
Sep 19 09:29:17.250675 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.250570    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-vwldn/must-gather-t64nw"]
Sep 19 09:29:17.251142 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.250796    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-vwldn/must-gather-t64nw" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" containerName="copy" containerID="cri-o://555098ca7cf6757601917ecffd38fa064b34e574839dcaf80bda18cb27acaaef" gracePeriod=2
Sep 19 09:29:17.253973 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:29:17.253923    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t64nw\" is forbidden: User \"system:node:ip-10-0-131-33.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vwldn\": no relationship found between node 'ip-10-0-131-33.ec2.internal' and this object" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:29:17.255930 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.255901    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-vwldn/must-gather-t64nw"]
Sep 19 09:29:17.554192 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.553967    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vwldn_must-gather-t64nw_d5765d31-65b8-4a7d-8132-cf9a36baa5ea/copy/0.log"
Sep 19 09:29:17.559017 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.558819    2577 generic.go:356] "Generic (PLEG): container finished" podID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" containerID="555098ca7cf6757601917ecffd38fa064b34e574839dcaf80bda18cb27acaaef" exitCode=143
Sep 19 09:29:17.640518 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.640493    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vwldn_must-gather-t64nw_d5765d31-65b8-4a7d-8132-cf9a36baa5ea/copy/0.log"
Sep 19 09:29:17.647535 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.647510    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:29:17.649736 ip-10-0-131-33 kubenswrapper[2577]: E0919 09:29:17.649699    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t64nw\" is forbidden: User \"system:node:ip-10-0-131-33.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vwldn\": no relationship found between node 'ip-10-0-131-33.ec2.internal' and this object" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:29:17.686957 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.686342    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-must-gather-output\") pod \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\" (UID: \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\") "
Sep 19 09:29:17.686957 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.686459    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmwgz\" (UniqueName: \"kubernetes.io/projected/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-kube-api-access-fmwgz\") pod \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\" (UID: \"d5765d31-65b8-4a7d-8132-cf9a36baa5ea\") "
Sep 19 09:29:17.689947 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.689897    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-kube-api-access-fmwgz" (OuterVolumeSpecName: "kube-api-access-fmwgz") pod "d5765d31-65b8-4a7d-8132-cf9a36baa5ea" (UID: "d5765d31-65b8-4a7d-8132-cf9a36baa5ea"). InnerVolumeSpecName "kube-api-access-fmwgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:29:17.697380 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.697245    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d5765d31-65b8-4a7d-8132-cf9a36baa5ea" (UID: "d5765d31-65b8-4a7d-8132-cf9a36baa5ea"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:29:17.787559 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.787422    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-must-gather-output\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:29:17.787559 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:17.787460    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmwgz\" (UniqueName: \"kubernetes.io/projected/d5765d31-65b8-4a7d-8132-cf9a36baa5ea-kube-api-access-fmwgz\") on node \"ip-10-0-131-33.ec2.internal\" DevicePath \"\""
Sep 19 09:29:18.510614 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:18.508400    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" path="/var/lib/kubelet/pods/d5765d31-65b8-4a7d-8132-cf9a36baa5ea/volumes"
Sep 19 09:29:18.569937 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:18.569910    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vwldn_must-gather-t64nw_d5765d31-65b8-4a7d-8132-cf9a36baa5ea/copy/0.log"
Sep 19 09:29:18.570757 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:18.570731    2577 scope.go:117] "RemoveContainer" containerID="555098ca7cf6757601917ecffd38fa064b34e574839dcaf80bda18cb27acaaef"
Sep 19 09:29:18.571515 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:18.571497    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vwldn/must-gather-t64nw"
Sep 19 09:29:18.597692 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:18.597657    2577 scope.go:117] "RemoveContainer" containerID="3b03f8700c461c6f0cc540b0e71957b274f9b0f0930cdda5121b1a0e2d79b711"
Sep 19 09:29:20.165315 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:20.165275    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-9jnzl_b7f4c415-703f-4079-a006-5851c37d5491/manager/0.log"
Sep 19 09:29:20.360104 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:20.360076    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6zhpz_21f44ad0-93f6-45c6-8d2a-f36c85490def/limitador/0.log"
Sep 19 09:29:20.392944 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:20.392915    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-trd7d_71ac61a2-7f18-42cf-b280-e37a2f2e24e1/manager/0.log"
Sep 19 09:29:21.855102 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:21.855068    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-xw5l6_e616fb2c-ac98-4a0b-8fcc-4f0b29fbcb36/cluster-monitoring-operator/0.log"
Sep 19 09:29:21.883549 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:21.883515    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-dkd42_6be1325b-dc1c-49cd-bc59-f3d0a92fbadb/kube-state-metrics/0.log"
Sep 19 09:29:21.899291 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:21.899245    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-dkd42_6be1325b-dc1c-49cd-bc59-f3d0a92fbadb/kube-rbac-proxy-main/0.log"
Sep 19 09:29:21.920827 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:21.920793    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-dkd42_6be1325b-dc1c-49cd-bc59-f3d0a92fbadb/kube-rbac-proxy-self/0.log"
Sep 19 09:29:21.947244 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:21.947191    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b996944bf-762gj_3c5550a3-16b8-4853-823e-619127563a54/metrics-server/0.log"
Sep 19 09:29:22.153519 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:22.153416    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qbs7v_e7657b45-d17a-41ae-8dc6-8e5e221aa1cf/node-exporter/0.log"
Sep 19 09:29:22.176134 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:22.176074    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qbs7v_e7657b45-d17a-41ae-8dc6-8e5e221aa1cf/kube-rbac-proxy/0.log"
Sep 19 09:29:22.195188 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:22.195163    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qbs7v_e7657b45-d17a-41ae-8dc6-8e5e221aa1cf/init-textfile/0.log"
Sep 19 09:29:22.220399 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:22.220339    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-7ftpc_251868a1-d2a6-42d9-b51a-1654c517d3d8/kube-rbac-proxy-main/0.log"
Sep 19 09:29:22.240301 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:22.240267    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-7ftpc_251868a1-d2a6-42d9-b51a-1654c517d3d8/kube-rbac-proxy-self/0.log"
Sep 19 09:29:22.260796 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:22.260742    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-7ftpc_251868a1-d2a6-42d9-b51a-1654c517d3d8/openshift-state-metrics/0.log"
Sep 19 09:29:24.064651 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:24.064616    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-ptbj8_e873fa1c-aba8-41f9-9f87-66b3b4439306/networking-console-plugin/0.log"
Sep 19 09:29:25.223693 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.223656    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"]
Sep 19 09:29:25.227618 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.224187    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" containerName="copy"
Sep 19 09:29:25.227618 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.224203    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" containerName="copy"
Sep 19 09:29:25.227618 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.224223    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" containerName="gather"
Sep 19 09:29:25.227618 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.224229    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" containerName="gather"
Sep 19 09:29:25.227618 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.224292    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" containerName="gather"
Sep 19 09:29:25.227618 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.224302    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5765d31-65b8-4a7d-8132-cf9a36baa5ea" containerName="copy"
Sep 19 09:29:25.228519 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.228498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.238324 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.238301    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"]
Sep 19 09:29:25.263865 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.263828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-sys\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.264037 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.263899    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-podres\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.264037 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.263939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-proc\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.264037 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.263974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-lib-modules\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.264037 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.263999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8j8\" (UniqueName: \"kubernetes.io/projected/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-kube-api-access-nk8j8\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.364771 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.364735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-proc\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.364771 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.364777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-lib-modules\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.365038 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.364796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8j8\" (UniqueName: \"kubernetes.io/projected/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-kube-api-access-nk8j8\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.365038 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.364870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-proc\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.365038 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.364882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-sys\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.365038 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.364929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-sys\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.365038 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.364942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-podres\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.365038 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.364958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-lib-modules\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.365394 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.365092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-podres\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.372841 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.372796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8j8\" (UniqueName: \"kubernetes.io/projected/b60ccc0d-676d-4ea7-a463-44a6c12cd10a-kube-api-access-nk8j8\") pod \"perf-node-gather-daemonset-rp2gc\" (UID: \"b60ccc0d-676d-4ea7-a463-44a6c12cd10a\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.541506 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.541431    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:25.829741 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:25.829707    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"]
Sep 19 09:29:25.831376 ip-10-0-131-33 kubenswrapper[2577]: W0919 09:29:25.831331    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb60ccc0d_676d_4ea7_a463_44a6c12cd10a.slice/crio-dd3e3528388f5a529a2e7ff31ce3954031bd163a03cb8c045c00afef2ad2b407 WatchSource:0}: Error finding container dd3e3528388f5a529a2e7ff31ce3954031bd163a03cb8c045c00afef2ad2b407: Status 404 returned error can't find the container with id dd3e3528388f5a529a2e7ff31ce3954031bd163a03cb8c045c00afef2ad2b407
Sep 19 09:29:26.596795 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:26.596762    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kdt58_28834fd2-afab-47de-a085-52668ee84b6f/dns/0.log"
Sep 19 09:29:26.616354 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:26.616323    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kdt58_28834fd2-afab-47de-a085-52668ee84b6f/kube-rbac-proxy/0.log"
Sep 19 09:29:26.649623 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:26.649587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc" event={"ID":"b60ccc0d-676d-4ea7-a463-44a6c12cd10a","Type":"ContainerStarted","Data":"51f8d69b064e39b6e88479dee3a84712608268f53142907200b84d86a6f750cb"}
Sep 19 09:29:26.649794 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:26.649645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc" event={"ID":"b60ccc0d-676d-4ea7-a463-44a6c12cd10a","Type":"ContainerStarted","Data":"dd3e3528388f5a529a2e7ff31ce3954031bd163a03cb8c045c00afef2ad2b407"}
Sep 19 09:29:26.649794 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:26.649708    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:26.665816 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:26.665754    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc" podStartSLOduration=1.6657378440000001 podStartE2EDuration="1.665737844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 09:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:29:26.664298725 +0000 UTC m=+4134.345882157" watchObservedRunningTime="2026-09-19 09:29:26.665737844 +0000 UTC m=+4134.347321274"
Sep 19 09:29:26.705035 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:26.705008    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ltmhg_a791e266-0859-46a0-920f-dcec31e7a326/dns-node-resolver/0.log"
Sep 19 09:29:27.241752 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:27.241720    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzm5f_c1c234fa-2c7e-43db-9f11-be533024f6a7/node-ca/0.log"
Sep 19 09:29:28.244331 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:28.244299    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/0.log"
Sep 19 09:29:28.247530 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:28.247503    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/1.log"
Sep 19 09:29:28.271392 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:28.271347    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f9d67fdf8-x2dbm_ddfd0060-8f85-422c-85c1-9570edc61e0d/router/0.log"
Sep 19 09:29:28.773565 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:28.773532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5ld78_be02c1ed-a740-42d1-be31-3290295ef972/serve-healthcheck-canary/0.log"
Sep 19 09:29:29.353833 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:29.353804    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9ldsl_2272806d-5340-4bd6-b669-8f5f2a74fb64/kube-rbac-proxy/0.log"
Sep 19 09:29:29.375226 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:29.375198    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9ldsl_2272806d-5340-4bd6-b669-8f5f2a74fb64/exporter/0.log"
Sep 19 09:29:29.400453 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:29.400427    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9ldsl_2272806d-5340-4bd6-b669-8f5f2a74fb64/extractor/0.log"
Sep 19 09:29:32.672615 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:32.672579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-rp2gc"
Sep 19 09:29:33.094824 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:33.094717    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7dcf4bff6-7f7nl_04c6aeb1-8f45-4595-a064-c3888a62c0a3/manager/0.log"
Sep 19 09:29:33.472581 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:33.472543    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6888f9f64c-f28sc_3479dbd7-c751-4d4a-9e75-985e0a7c3c15/manager/0.log"
Sep 19 09:29:33.493636 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:33.493607    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-2wjrp_3c99ff65-cffa-4499-b71a-a418c9b2f7bd/server/0.log"
Sep 19 09:29:34.254336 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:34.254231    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-f76dd_3a837fe5-4583-41b4-828f-6b3bc09a1354/manager/0.log"
Sep 19 09:29:34.293491 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:34.293461    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9kttj_8a33e212-a14a-455d-b751-e7ae18d8a7fa/s3-init/0.log"
Sep 19 09:29:34.309256 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:34.309221    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-9kttj_8a33e212-a14a-455d-b751-e7ae18d8a7fa/download-hf-model/0.log"
Sep 19 09:29:34.425870 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:34.425824    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/0.log"
Sep 19 09:29:34.429288 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:34.429172    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/1.log"
Sep 19 09:29:34.448149 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:34.448122    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f9d67fdf8-x2dbm_ddfd0060-8f85-422c-85c1-9570edc61e0d/router/0.log"
Sep 19 09:29:35.606874 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:35.606842    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/0.log"
Sep 19 09:29:35.610634 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:35.610605    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-v2dfl_f6b1ce6b-ad39-472b-bdc5-ee4bb2b76a67/istio-proxy/1.log"
Sep 19 09:29:35.637487 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:35.637460    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f9d67fdf8-x2dbm_ddfd0060-8f85-422c-85c1-9570edc61e0d/router/0.log"
Sep 19 09:29:39.566557 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:39.566523    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-48wml_dde16ec8-2ee3-48a3-a755-b8d40692076b/kube-storage-version-migrator-operator/1.log"
Sep 19 09:29:39.567446 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:39.567427    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-48wml_dde16ec8-2ee3-48a3-a755-b8d40692076b/kube-storage-version-migrator-operator/0.log"
Sep 19 09:29:40.627114 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:40.627087    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fh57_a0649707-f033-46a5-b1ea-595b2a403f01/kube-multus-additional-cni-plugins/0.log"
Sep 19 09:29:40.646194 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:40.646172    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fh57_a0649707-f033-46a5-b1ea-595b2a403f01/egress-router-binary-copy/0.log"
Sep 19 09:29:40.666557 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:40.666538    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fh57_a0649707-f033-46a5-b1ea-595b2a403f01/cni-plugins/0.log"
Sep 19 09:29:40.684948 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:40.684931    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fh57_a0649707-f033-46a5-b1ea-595b2a403f01/bond-cni-plugin/0.log"
Sep 19 09:29:40.703556 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:40.703539    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fh57_a0649707-f033-46a5-b1ea-595b2a403f01/routeoverride-cni/0.log"
Sep 19 09:29:40.722490 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:40.722464    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fh57_a0649707-f033-46a5-b1ea-595b2a403f01/whereabouts-cni-bincopy/0.log"
Sep 19 09:29:40.741647 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:40.741625    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6fh57_a0649707-f033-46a5-b1ea-595b2a403f01/whereabouts-cni/0.log"
Sep 19 09:29:40.929345 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:40.929274    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qr49j_4e3c0738-763f-42cc-b72c-f90bd30e1255/kube-multus/0.log"
Sep 19 09:29:41.120802 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:41.120769    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kwpz5_5cbb75e1-d58d-41d4-ba14-9e3194e793f8/network-metrics-daemon/0.log"
Sep 19 09:29:41.138740 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:41.138713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kwpz5_5cbb75e1-d58d-41d4-ba14-9e3194e793f8/kube-rbac-proxy/0.log"
Sep 19 09:29:41.974381 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:41.974345    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-controller/0.log"
Sep 19 09:29:41.990608 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:41.990575    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/0.log"
Sep 19 09:29:41.990608 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:41.990572    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovn-acl-logging/1.log"
Sep 19 09:29:42.008610 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:42.008586    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/kube-rbac-proxy-node/0.log"
Sep 19 09:29:42.028162 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:42.028135    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/kube-rbac-proxy-ovn-metrics/0.log"
Sep 19 09:29:42.045378 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:42.045344    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/northd/0.log"
Sep 19 09:29:42.065101 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:42.065076    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/nbdb/0.log"
Sep 19 09:29:42.084173 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:42.084155    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/sbdb/0.log"
Sep 19 09:29:42.150804 ip-10-0-131-33 kubenswrapper[2577]: I0919 09:29:42.150773    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6np9d_fdb1d57e-d87d-4155-b821-b40189f0e04d/ovnkube-controller/0.log"