Sep 09 19:54:50.101985 ip-10-0-138-141 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 19:54:50.509153 ip-10-0-138-141 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:54:50.509153 ip-10-0-138-141 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 19:54:50.509153 ip-10-0-138-141 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:54:50.509153 ip-10-0-138-141 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 19:54:50.509153 ip-10-0-138-141 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:54:50.511204 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.511085    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 19:54:50.515032 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515015    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:50.515032 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515032    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515036    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515039    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515043    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515046    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515049    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515053    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515056    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515059    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515062    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515067    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515071    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515074    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515077    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515081    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515084    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515087    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515090    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515093    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:50.515102 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515096    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515098    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515101    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515103    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515106    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515110    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515114    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515117    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515119    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515122    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515125    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515127    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515130    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515133    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515135    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515138    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515141    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515143    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515146    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:50.515558 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515148    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515151    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515154    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515157    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515159    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515162    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515164    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515170    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515172    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515175    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515178    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515181    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515183    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515186    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515189    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515193    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515197    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515200    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515203    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515206    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:50.516076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515209    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515211    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515214    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515217    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515219    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515222    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515224    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515227    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515230    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515232    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515235    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515237    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515241    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515243    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515246    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515249    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515251    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515254    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515256    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515259    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:50.516565 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515262    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515264    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515267    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515269    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515272    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515275    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515279    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515282    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515286    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515288    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515291    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515293    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515296    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515299    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515301    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515304    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515306    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515309    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515311    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.515314    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:50.517126 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516041    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516048    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516051    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516054    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516056    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516059    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516062    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516064    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516067    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516070    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516072    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516075    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516077    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516081    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516084    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516087    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516090    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516093    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516095    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516098    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:50.517635 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516101    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516103    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516106    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516108    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516111    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516114    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516116    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516119    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516121    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516123    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516126    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516128    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516131    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516133    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516136    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516138    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516141    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516143    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516148    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:50.518160 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516151    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516153    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516156    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516158    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516160    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516164    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516167    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516170    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516173    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516175    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516178    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516180    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516184    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516186    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516189    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516191    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516194    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516196    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516199    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516202    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:50.518625 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516205    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516209    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516213    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516217    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516220    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516223    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516226    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516229    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516232    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516234    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516237    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516239    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516242    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516244    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516248    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516250    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516253    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516256    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516258    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:50.519141 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516262    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516265    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516268    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516271    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516273    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516276    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516278    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516281    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516283    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516286    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516289    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516291    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516294    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516296    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516299    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516301    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516304    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516307    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516309    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516312    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:50.519660 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516315    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516697    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516702    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516705    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516722    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516725    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516728    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516730    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516734    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516737    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516740    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516742    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516745    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516748    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516751    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516754    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516757    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516759    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516762    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516765    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:50.520328 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516767    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516770    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516772    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516775    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516777    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516780    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516782    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516785    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516787    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516790    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516792    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516795    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516797    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516800    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516802    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516805    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516807    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516809    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516812    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516815    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:50.521035 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516818    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516821    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516824    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516826    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516829    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516833    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516837    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516839    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516842    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516845    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516847    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516850    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516854    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516858    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516861    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516863    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516866    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516869    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516872    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:50.521608 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516875    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516878    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516881    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516884    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516888    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516894    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516897    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516899    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516902    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516904    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516907    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516909    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516912    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516915    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516917    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516920    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516923    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516925    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516928    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:50.522136 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516931    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516935    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516939    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516942    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516945    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516948    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516950    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516953    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516955    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516958    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516960    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516963    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516966    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516968    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516971    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516973    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516976    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516979    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516983    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:50.522652 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516985    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516988    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.516990    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517080    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517120    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517127    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517133    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517138    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517142    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517146    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517151    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517155    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517158    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517161    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517164    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517168    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517171    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517174    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517177    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517179    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517182    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517186    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517189    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517192    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 19:54:50.523218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517211    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517216    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517220    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517223    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517227    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517230    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517234    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517237    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517240    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517243    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517246    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517250    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517253    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517256    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517261    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517264    2578 flags.go:64] FLAG: --enable-server="true"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517267    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517272    2578 flags.go:64] FLAG: --event-burst="100"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517275    2578 flags.go:64] FLAG: --event-qps="50"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517278    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517281    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517284    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517287    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517290    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517294    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 19:54:50.523865 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517298    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517301    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517304    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517307    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517309    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517312    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517315    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517318    2578 flags.go:64] FLAG: --feature-gates=""
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517322    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517325    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517328    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517331    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517334    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517337    2578 flags.go:64] FLAG: --help="false"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517341    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517344    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517347    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517350    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517353    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517356    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517359    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517364    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517367    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 19:54:50.524515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517370    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517373    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517376    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517379    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517382    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517385    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517388    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517390    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517393    2578 flags.go:64] FLAG: --lock-file=""
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517397    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517400    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517403    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517408    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517411    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517414    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517417    2578 flags.go:64] FLAG: --logging-format="text"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517419    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517423    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517425    2578 flags.go:64] FLAG: --manifest-url=""
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517428    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517435    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517438    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517442    2578 flags.go:64] FLAG: --max-pods="110"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517446    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517449    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 19:54:50.525148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517451    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517454    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517457    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517460    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517463    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517472    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517476    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517479    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517482    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517485    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517490    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517493    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517496    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517499    2578 flags.go:64] FLAG: --port="10250"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517503    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517505    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8bb7eff24585f43"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517508    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517512    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517515    2578 flags.go:64] FLAG: --register-node="true"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517518    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517522    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517525    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517528    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517530    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517534    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 19:54:50.525777 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517537    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517540    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517543    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517546    2578 flags.go:64] FLAG: --runonce="false"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517549    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517552    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517555    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517558    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517561    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517564    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517567    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517570    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517573    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517577    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517580    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517583    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517586    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517589    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517592    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517597    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517600    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517603    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517607    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517610    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517612    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 19:54:50.526404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517616    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517619    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517622    2578 flags.go:64] FLAG: --v="2"
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517626    2578 flags.go:64] FLAG: --version="false"
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517630    2578 flags.go:64] FLAG: --vmodule=""
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517634    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.517638    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517771    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517776    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517780    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517783    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517786    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517791    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517794    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517797    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517800    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517802    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517805    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517808    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517810    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517814    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:50.527105 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517817    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517820    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517822    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517825    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517828    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517832    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517836    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517839    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517842    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517845    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517848    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517851    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517854    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517857    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517860    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517863    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517865    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517868    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517870    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517873    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:50.527629 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517875    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517878    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517880    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517885    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517887    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517890    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517892    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517895    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517898    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517900    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517903    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517908    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517913    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517916    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517918    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517921    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517923    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517926    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517929    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517931    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:50.528175 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517934    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517936    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517939    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517942    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517944    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517947    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517950    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517953    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517955    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517958    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517960    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517963    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517965    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517968    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517970    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517974    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517977    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517980    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517982    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517985    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:50.528772 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517987    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517990    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.517992    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518004    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518007    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518010    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518012    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518015    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518018    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518020    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518024    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518028    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518031    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518034    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518037    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518040    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518043    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518046    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518049    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:50.529281 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518052    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:50.529795 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518055    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:50.529795 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518057    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:50.529795 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518060    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:50.529795 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518062    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:50.529795 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.518065    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:50.529795 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.518868    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:54:50.529795 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.529008    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 19:54:50.529795 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.529025    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 19:54:50.530009 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.529976    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 19:54:50.530009 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.529984    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 19:54:50.530062 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530043    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:50.530062 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530048    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:50.530062 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530052    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:50.530062 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530055    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:50.530062 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530058    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:50.530062 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530061    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:50.530062 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530064    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530067    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530070    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530073    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530075    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530078    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530081    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530084    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530086    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530089    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530091    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530095    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530097    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530100    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530103    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530106    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530108    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530111    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530114    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530116    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:50.530233 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530119    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530122    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530124    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530127    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530129    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530132    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530135    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530137    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530140    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530143    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530145    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530148    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530150    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530153    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530156    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530159    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530162    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530164    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530167    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:50.530742 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530169    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530173    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530176    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530179    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530182    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530184    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530187    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530190    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530193    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530195    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530198    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530201    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530204    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530206    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530209    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530211    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530214    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530217    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530219    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530222    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:50.531211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530225    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530227    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530230    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530232    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530235    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530238    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530241    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530244    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530247    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530250    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530252    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530255    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530258    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530260    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530264    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530267    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530270    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530272    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530275    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530278    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:50.531690 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530280    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530283    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530286    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530288    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530291    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530293    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530296    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530300    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530305    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530307    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530310    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530314    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530318    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530321    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:50.532247 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.530327    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530441    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530446    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530449    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530452    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530456    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530460    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530464    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530468    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530471    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530474    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530476    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530479    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530482    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530484    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530487    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530490    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530492    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530495    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:50.532588 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530497    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530500    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530503    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530505    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530508    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530510    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530513    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530515    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530518    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530520    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530523    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530526    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530528    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530531    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530534    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530536    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530538    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530541    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530543    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530546    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:50.533076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530549    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530552    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530554    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530557    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530560    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530562    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530565    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530568    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530571    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530573    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530576    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530578    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530581    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530583    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530586    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530588    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530591    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530593    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530596    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530599    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:50.533577 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530601    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530604    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530607    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530609    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530612    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530614    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530617    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530619    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530622    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530625    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530628    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530631    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530633    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530636    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530638    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530641    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530644    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530646    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530648    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530651    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:50.534051 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530655    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530657    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530660    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530662    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530665    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530667    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530671    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530675    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530678    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530681    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530683    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530686    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530688    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530691    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530694    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530696    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530699    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530702    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530705    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:50.534556 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530721    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:50.535076 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:50.530724    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:50.535076 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.530729    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:54:50.535076 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.530851    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 19:54:50.535076 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.532906    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 19:54:50.535076 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.533977    2578 server.go:1019] "Starting client certificate rotation"
Sep 09 19:54:50.535076 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.534075    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:54:50.535076 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.534674    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:54:50.537113 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.537098    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:54:50.542298 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.542168    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:54:50.553435 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.553414    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 09 19:54:50.556385 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.556370    2578 log.go:25] "Validated CRI v1 image API"
Sep 09 19:54:50.557261 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.557247    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 19:54:50.560592 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.560570    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9e82677e-036b-4690-a713-da06f0fa1116:/dev/nvme0n1p3 d1ceb4cd-e2f1-4b73-8438-f9a604849230:/dev/nvme0n1p4]
Sep 09 19:54:50.560657 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.560590    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 19:54:50.564042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.564024    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:54:50.566577 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.566466    2578 manager.go:216] Machine: {Timestamp:2026-09-09 19:54:50.564122585 +0000 UTC m=+0.346374242 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098092 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2366752f80bc6e11fa929a336ba594 SystemUUID:ec236675-2f80-bc6e-11fa-929a336ba594 BootID:4a27c7f9-2b71-4bda-8e31-ec2632dacaca Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:29:75:32:ef:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:29:75:32:ef:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:9d:f9:68:11:7f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 19:54:50.566577 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.566575    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 19:54:50.566700 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.566688    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 19:54:50.569799 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.569775    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 19:54:50.569932 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.569801    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-141.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 19:54:50.571178 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.571168    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 19:54:50.571211 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.571181    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 19:54:50.571211 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.571189    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:54:50.571857 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.571846    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:54:50.573091 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.573081    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 19:54:50.576913 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.576902    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:54:50.577028 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.577019    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 19:54:50.577065 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.577047    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 19:54:50.577099 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.577064    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 19:54:50.577099 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.577080    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 19:54:50.577099 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.577089    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 09 19:54:50.577099 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.577098    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 19:54:50.577451 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.577439    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:54:50.577493 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.577458    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:54:50.579700 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.579683    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 19:54:50.581941 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.581927    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 19:54:50.581980 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.581958    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 19:54:50.582111 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582101    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 19:54:50.582149 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582116    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 19:54:50.582149 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582122    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 19:54:50.582149 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582128    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 19:54:50.582149 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582136    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 19:54:50.582149 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582141    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 19:54:50.582640 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582632    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 19:54:50.582670 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582648    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 19:54:50.582670 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582655    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 19:54:50.582670 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582660    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 19:54:50.582769 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582673    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 19:54:50.582769 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.582680    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 19:54:50.583594 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.583581    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 19:54:50.583634 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.583595    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 19:54:50.587286 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.587272    2578 server.go:1268] "Started kubelet"
Sep 09 19:54:50.587396 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.587363    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 19:54:50.587493 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.587370    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 19:54:50.587493 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.587437    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 19:54:50.588149 ip-10-0-138-141 systemd[1]: Started Kubernetes Kubelet.
Sep 09 19:54:50.588657 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.588641    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 19:54:50.588866 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.588845    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-141.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 19:54:50.589001 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.588979    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 19:54:50.589090 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.589010    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-141.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 19:54:50.590984 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.590114    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 19:54:50.591980 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.591962    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 19:54:50.594166 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.592932    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-141.ec2.internal.18d3bf302a6b8b23  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-141.ec2.internal,UID:ip-10-0-138-141.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-141.ec2.internal,},FirstTimestamp:2026-09-09 19:54:50.587245347 +0000 UTC m=+0.369497008,LastTimestamp:2026-09-09 19:54:50.587245347 +0000 UTC m=+0.369497008,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-141.ec2.internal,}"
Sep 09 19:54:50.595277 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.595256    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 19:54:50.595453 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.595340    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 19:54:50.597535 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.595825    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:50.597535 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.596263    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 19:54:50.598159 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.597904    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 19:54:50.598159 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.597924    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 19:54:50.598159 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.598064    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 19:54:50.598159 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.598071    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 19:54:50.599332 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.599314    2578 factory.go:55] Registering systemd factory
Sep 09 19:54:50.599622 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.599597    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 09 19:54:50.599853 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.599838    2578 factory.go:153] Registering CRI-O factory
Sep 09 19:54:50.599932 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.599856    2578 factory.go:223] Registration of the crio container factory successfully
Sep 09 19:54:50.599932 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.599903    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 19:54:50.599932 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.599924    2578 factory.go:103] Registering Raw factory
Sep 09 19:54:50.600087 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.599936    2578 manager.go:1196] Started watching for new ooms in manager
Sep 09 19:54:50.600383 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.600294    2578 manager.go:349] Starting recovery of all containers
Sep 09 19:54:50.607851 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.607824    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-141.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 19:54:50.607851 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.607840    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 19:54:50.610329 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.610316    2578 manager.go:354] Recovery completed
Sep 09 19:54:50.612952 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.612922    2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 19:54:50.617142 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.617128    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:50.619094 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.619074    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:50.619176 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.619107    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:50.619176 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.619118    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:50.619729 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.619698    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 19:54:50.619729 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.619726    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 19:54:50.619847 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.619745    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:54:50.619847 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.619751    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:50.620478 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.620459    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:50.620536 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.620486    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:50.620536 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.620501    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:50.621926 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.621914    2578 policy_none.go:49] "None policy: Start"
Sep 09 19:54:50.621970 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.621930    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 19:54:50.621970 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.621941    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 19:54:50.623611 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.623596    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vdxg6"
Sep 09 19:54:50.623978 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.623967    2578 policy_none.go:47] "Start"
Sep 09 19:54:50.629213 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.629111    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-141.ec2.internal.18d3bf302c518134  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-141.ec2.internal,UID:ip-10-0-138-141.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-141.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-141.ec2.internal,},FirstTimestamp:2026-09-09 19:54:50.6190933 +0000 UTC m=+0.401344957,LastTimestamp:2026-09-09 19:54:50.6190933 +0000 UTC m=+0.401344957,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-141.ec2.internal,}"
Sep 09 19:54:50.632234 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.632214    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vdxg6"
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.663313    2578 manager.go:337] "Starting Device Plugin manager"
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.663362    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.663376    2578 server.go:87] "Starting device plugin registration server"
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.663827    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.663841    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.663992    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.664084    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.664092    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.664586    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 19:54:50.671836 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.664631    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:50.695004 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.694974    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 19:54:50.696227 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.696199    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 19:54:50.696361 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.696248    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 19:54:50.696361 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.696271    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 19:54:50.696361 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.696316    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 19:54:50.699049 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.699024    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:50.764887 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.764778    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:50.765843 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.765824    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:50.765981 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.765854    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:50.765981 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.765867    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:50.765981 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.765898    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.775332 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.775315    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.775387 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.775344    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-141.ec2.internal\": node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:50.797088 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.797053    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal"]
Sep 09 19:54:50.797207 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.797127    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:50.797968 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.797954    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:50.798028 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.797982    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:50.798028 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.797993    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:50.800242 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.800229    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:50.800377 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.800362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.800457 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.800425    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:50.800967 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.800952    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:50.800967 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.800966    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:50.801077 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.800982    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:50.801077 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.800986    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:50.801077 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.800996    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:50.801077 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.800997    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:50.803114 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.803097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.803214 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.803126    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:50.803797 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.803783    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:50.803885 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.803809    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:50.803885 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.803823    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:50.805001 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.804985    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:50.830425 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.830404    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-141.ec2.internal\" not found" node="ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.834681 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.834659    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-141.ec2.internal\" not found" node="ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.899723 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.899689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7836c45ac94a1983d60a015a961f0779-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal\" (UID: \"7836c45ac94a1983d60a015a961f0779\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.899826 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.899734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7836c45ac94a1983d60a015a961f0779-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal\" (UID: \"7836c45ac94a1983d60a015a961f0779\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.899826 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:50.899754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/181a392e75f0abd87761766443396815-config\") pod \"kube-apiserver-proxy-ip-10-0-138-141.ec2.internal\" (UID: \"181a392e75f0abd87761766443396815\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:50.905795 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:50.905773    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:51.000214 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:51.000182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7836c45ac94a1983d60a015a961f0779-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal\" (UID: \"7836c45ac94a1983d60a015a961f0779\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:51.000354 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:51.000134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7836c45ac94a1983d60a015a961f0779-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal\" (UID: \"7836c45ac94a1983d60a015a961f0779\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:51.000354 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:51.000245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7836c45ac94a1983d60a015a961f0779-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal\" (UID: \"7836c45ac94a1983d60a015a961f0779\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:51.000354 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:51.000261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/181a392e75f0abd87761766443396815-config\") pod \"kube-apiserver-proxy-ip-10-0-138-141.ec2.internal\" (UID: \"181a392e75f0abd87761766443396815\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:51.000354 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:51.000294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/181a392e75f0abd87761766443396815-config\") pod \"kube-apiserver-proxy-ip-10-0-138-141.ec2.internal\" (UID: \"181a392e75f0abd87761766443396815\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:51.000354 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:51.000346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7836c45ac94a1983d60a015a961f0779-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal\" (UID: \"7836c45ac94a1983d60a015a961f0779\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:51.006268 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:51.006222    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:51.107031 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:51.106959    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:51.133119 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:51.133097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:51.138190 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:51.137949    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:51.207685 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:51.207649    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:52.501737 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:52.501708    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:52.602185 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:52.602164    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-141.ec2.internal\" not found"
Sep 09 19:54:52.639395 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.639370    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:52.690762 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.690740    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:52.706507 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.706483    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:52.708339 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.708324    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:54:52.709432 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.709417    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal"
Sep 09 19:54:52.719249 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.719235    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:54:52.727429 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.727415    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 19:54:52.727550 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.727532    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:54:52.727594 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.727571    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:54:52.727594 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.727572    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:54:52.771722 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.771705    2578 apiserver.go:52] "Watching apiserver"
Sep 09 19:54:52.788950 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.788931    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 19:54:52.789287 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.789268    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-rxfgd","openshift-multus/multus-additional-cni-plugins-6b6dl","openshift-network-diagnostics/network-check-target-t7mhh","openshift-network-operator/iptables-alerter-vps6j","openshift-ovn-kubernetes/ovnkube-node-ccdvx","kube-system/global-pull-secret-syncer-2458j","kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l","openshift-cluster-node-tuning-operator/tuned-rj4vk","openshift-dns/node-resolver-ztrxp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal","openshift-multus/multus-k6jj7","openshift-multus/network-metrics-daemon-8r42k","kube-system/konnectivity-agent-jw9j8"]
Sep 09 19:54:52.789356 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.789293    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 19:54:52.791667 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.791651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:54:52.791760 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:52.791740    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:54:52.793614 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.793597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:52.795911 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.795893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:52.796167 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.796142    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 19:54:52.796258 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.796232    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:54:52.796335 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.796270    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rqbsj\""
Sep 09 19:54:52.796459 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.796294    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 19:54:52.796818 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.796761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:54:52.798613 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.798451    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 19:54:52.798714 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.798598    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 19:54:52.798771 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.798711    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lsb5l\""
Sep 09 19:54:52.798771 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.798711    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 19:54:52.798960 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.798898    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 19:54:52.799255 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.798992    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 19:54:52.799402 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.799387    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.801623 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.801607    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:52.801694 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:52.801665    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:54:52.802856 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.802839    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 19:54:52.802933 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.802852    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bc7pc\""
Sep 09 19:54:52.802933 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.802866    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 19:54:52.802933 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.802875    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 19:54:52.802933 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.802886    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 19:54:52.802933 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.802928    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 19:54:52.803120 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.802873    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 19:54:52.803795 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.803770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:52.804010 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.803895    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:54:52.805932 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.805910    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 19:54:52.806027 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.805952    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lrdct\""
Sep 09 19:54:52.806086 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.806045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:52.806394 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.806377    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 19:54:52.806479 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.806412    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 19:54:52.808521 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.808390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.808620 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.808555    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 19:54:52.808786 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.808769    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 19:54:52.808850 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.808823    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 19:54:52.808986 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.808973    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pmv9x\""
Sep 09 19:54:52.810714 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.810697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:52.811484 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.811448    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:54:52.811569 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.811521    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 19:54:52.811569 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.811566    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fqtvb\""
Sep 09 19:54:52.812915 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.812900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.813280 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.813251    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 19:54:52.813373 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.813326    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jzvvv\""
Sep 09 19:54:52.813988 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.813975    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 19:54:52.815057 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.815043    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 19:54:52.815526 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.815505    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rfvlk\""
Sep 09 19:54:52.815935 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.815921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:52.816021 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:52.816003    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:54:52.818099 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.818077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:54:52.820518 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.820504    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 19:54:52.820663 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.820649    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j6zfh\""
Sep 09 19:54:52.820735 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.820654    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 19:54:52.827466 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.827452    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ng4kd"
Sep 09 19:54:52.828514 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.828475    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 19:49:51 +0000 UTC" deadline="2028-05-19 03:32:39.199731806 +0000 UTC"
Sep 09 19:54:52.828557 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.828514    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14815h37m46.371219639s"
Sep 09 19:54:52.837175 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.837159    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ng4kd"
Sep 09 19:54:52.893058 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.892364    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 19:54:52.896942 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.896919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34db695b-0f6e-43c2-b92f-65214c6f28e4-kubelet-config\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:52.897042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.896948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:52.897042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.896968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-modprobe-d\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.897042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.896991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-systemd\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.897042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-socket-dir-parent\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.897042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:52.897042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-var-lib-cni-bin\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-var-lib-kubelet\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-systemd-units\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897115    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-run-ovn\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-node-log\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-conf-dir\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-system-cni-dir\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-run-openvswitch\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-socket-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:52.897267 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-registration-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qwn\" (UniqueName: \"kubernetes.io/projected/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-kube-api-access-f4qwn\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-cnibin\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-run-netns\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897346    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-hostroot\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-run-k8s-cni-cncf-io\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-run-systemd\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lgh9\" (UniqueName: \"kubernetes.io/projected/175a2b7e-f1c9-4a5c-bc4c-75f73cc79808-kube-api-access-9lgh9\") pod \"node-ca-rxfgd\" (UID: \"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808\") " pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4zn\" (UniqueName: \"kubernetes.io/projected/ee80103b-11e9-42f0-a4d2-345e094101a6-kube-api-access-2q4zn\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-slash\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-log-socket\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-cni-netd\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/375948f8-eecf-4ce7-a0ab-195b78a6d99f-ovn-node-metrics-cert\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4c2aac3-9593-4d44-9e55-92a0940bd662-tmp\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmrl\" (UniqueName: \"kubernetes.io/projected/d4c2aac3-9593-4d44-9e55-92a0940bd662-kube-api-access-jcmrl\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-system-cni-dir\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2b45f30-896c-434f-b5a5-029893e1fd1b-agent-certs\") pod \"konnectivity-agent-jw9j8\" (UID: \"c2b45f30-896c-434f-b5a5-029893e1fd1b\") " pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897823    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/375948f8-eecf-4ce7-a0ab-195b78a6d99f-env-overrides\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-sys-fs\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr88b\" (UniqueName: \"kubernetes.io/projected/cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6-kube-api-access-wr88b\") pod \"node-resolver-ztrxp\" (UID: \"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6\") " pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-daemon-config\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-etc-openvswitch\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9ts\" (UniqueName: \"kubernetes.io/projected/375948f8-eecf-4ce7-a0ab-195b78a6d99f-kube-api-access-5m9ts\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.897983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-kubernetes\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-sysctl-conf\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-cni-bin\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.898257 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7733c39-04f6-427d-bda9-9071b83f615c-host-slash\") pod \"iptables-alerter-vps6j\" (UID: \"b7733c39-04f6-427d-bda9-9071b83f615c\") " pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrn6x\" (UniqueName: \"kubernetes.io/projected/b7733c39-04f6-427d-bda9-9071b83f615c-kube-api-access-vrn6x\") pod \"iptables-alerter-vps6j\" (UID: \"b7733c39-04f6-427d-bda9-9071b83f615c\") " pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee80103b-11e9-42f0-a4d2-345e094101a6-cni-binary-copy\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-run-multus-certs\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-etc-kubernetes\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6-hosts-file\") pod \"node-resolver-ztrxp\" (UID: \"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6\") " pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-cni-dir\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2b45f30-896c-434f-b5a5-029893e1fd1b-konnectivity-ca\") pod \"konnectivity-agent-jw9j8\" (UID: \"c2b45f30-896c-434f-b5a5-029893e1fd1b\") " pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdgft\" (UniqueName: \"kubernetes.io/projected/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-kube-api-access-rdgft\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6-tmp-dir\") pod \"node-resolver-ztrxp\" (UID: \"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6\") " pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn82w\" (UniqueName: \"kubernetes.io/projected/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-kube-api-access-dn82w\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-kubelet\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-var-lib-openvswitch\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/375948f8-eecf-4ce7-a0ab-195b78a6d99f-ovnkube-config\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-sysctl-d\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-host\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.898943 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-var-lib-cni-multus\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-cnibin\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/175a2b7e-f1c9-4a5c-bc4c-75f73cc79808-host\") pod \"node-ca-rxfgd\" (UID: \"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808\") " pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/175a2b7e-f1c9-4a5c-bc4c-75f73cc79808-serviceca\") pod \"node-ca-rxfgd\" (UID: \"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808\") " pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-device-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-run\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7733c39-04f6-427d-bda9-9071b83f615c-iptables-alerter-script\") pod \"iptables-alerter-vps6j\" (UID: \"b7733c39-04f6-427d-bda9-9071b83f615c\") " pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-run-netns\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-lib-modules\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-var-lib-kubelet\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-tuned\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/375948f8-eecf-4ce7-a0ab-195b78a6d99f-ovnkube-script-lib\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34db695b-0f6e-43c2-b92f-65214c6f28e4-dbus\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-sysconfig\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.899542 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.898884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-sys\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:52.903773 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:52.903744    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:54:52.903879 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:52.903781    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:54:52.903879 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:52.903793    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x8vd5 for pod openshift-network-diagnostics/network-check-target-t7mhh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:52.903879 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:52.903859    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5 podName:bc10e010-ba02-48e7-acec-96938127af83 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:53.403839312 +0000 UTC m=+1.992492967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x8vd5" (UniqueName: "kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5") pod "network-check-target-t7mhh" (UID: "bc10e010-ba02-48e7-acec-96938127af83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:52.999830 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.999625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-run-netns\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.999840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-lib-modules\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.999860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-var-lib-kubelet\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.999874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-tuned\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.999897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.999728    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-run-netns\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.999931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/375948f8-eecf-4ce7-a0ab-195b78a6d99f-ovnkube-script-lib\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.999969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-var-lib-kubelet\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:52.999997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34db695b-0f6e-43c2-b92f-65214c6f28e4-dbus\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-sysconfig\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-sys\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34db695b-0f6e-43c2-b92f-65214c6f28e4-kubelet-config\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-lib-modules\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-sysconfig\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-modprobe-d\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-sys\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-systemd\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/34db695b-0f6e-43c2-b92f-65214c6f28e4-dbus\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-socket-dir-parent\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000172    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/34db695b-0f6e-43c2-b92f-65214c6f28e4-kubelet-config\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-var-lib-cni-bin\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-systemd\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.000238    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-modprobe-d\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-var-lib-cni-bin\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-socket-dir-parent\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.000494 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-var-lib-kubelet\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-systemd-units\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-run-ovn\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-var-lib-kubelet\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.000395    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret podName:34db695b-0f6e-43c2-b92f-65214c6f28e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:53.500374049 +0000 UTC m=+2.089027727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret") pod "global-pull-secret-syncer-2458j" (UID: "34db695b-0f6e-43c2-b92f-65214c6f28e4") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-run-ovn\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-systemd-units\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-node-log\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-conf-dir\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000539    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-system-cni-dir\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-run-openvswitch\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-conf-dir\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-node-log\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/375948f8-eecf-4ce7-a0ab-195b78a6d99f-ovnkube-script-lib\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-run-openvswitch\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-system-cni-dir\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-socket-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000650    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-registration-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qwn\" (UniqueName: \"kubernetes.io/projected/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-kube-api-access-f4qwn\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-registration-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-socket-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-cnibin\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-run-netns\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-hostroot\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-hostroot\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-cnibin\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.000993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-run-netns\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-run-k8s-cni-cncf-io\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.001923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-run-k8s-cni-cncf-io\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-run-systemd\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-run-systemd\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lgh9\" (UniqueName: \"kubernetes.io/projected/175a2b7e-f1c9-4a5c-bc4c-75f73cc79808-kube-api-access-9lgh9\") pod \"node-ca-rxfgd\" (UID: \"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808\") " pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001166    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4zn\" (UniqueName: \"kubernetes.io/projected/ee80103b-11e9-42f0-a4d2-345e094101a6-kube-api-access-2q4zn\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-slash\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-log-socket\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-cni-netd\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/375948f8-eecf-4ce7-a0ab-195b78a6d99f-ovn-node-metrics-cert\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001326    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4c2aac3-9593-4d44-9e55-92a0940bd662-tmp\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-slash\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmrl\" (UniqueName: \"kubernetes.io/projected/d4c2aac3-9593-4d44-9e55-92a0940bd662-kube-api-access-jcmrl\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-system-cni-dir\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-cni-netd\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-log-socket\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.002692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2b45f30-896c-434f-b5a5-029893e1fd1b-agent-certs\") pod \"konnectivity-agent-jw9j8\" (UID: \"c2b45f30-896c-434f-b5a5-029893e1fd1b\") " pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-system-cni-dir\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/375948f8-eecf-4ce7-a0ab-195b78a6d99f-env-overrides\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-sys-fs\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr88b\" (UniqueName: \"kubernetes.io/projected/cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6-kube-api-access-wr88b\") pod \"node-resolver-ztrxp\" (UID: \"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6\") " pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-sys-fs\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-daemon-config\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-etc-openvswitch\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001865    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9ts\" (UniqueName: \"kubernetes.io/projected/375948f8-eecf-4ce7-a0ab-195b78a6d99f-kube-api-access-5m9ts\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.001960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-etc-openvswitch\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-kubernetes\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-sysctl-conf\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-cni-bin\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7733c39-04f6-427d-bda9-9071b83f615c-host-slash\") pod \"iptables-alerter-vps6j\" (UID: \"b7733c39-04f6-427d-bda9-9071b83f615c\") " pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:53.003519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrn6x\" (UniqueName: \"kubernetes.io/projected/b7733c39-04f6-427d-bda9-9071b83f615c-kube-api-access-vrn6x\") pod \"iptables-alerter-vps6j\" (UID: \"b7733c39-04f6-427d-bda9-9071b83f615c\") " pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee80103b-11e9-42f0-a4d2-345e094101a6-cni-binary-copy\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-run-multus-certs\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-run-ovn-kubernetes\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002293    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-cni-bin\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002321    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-kubernetes\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b7733c39-04f6-427d-bda9-9071b83f615c-host-slash\") pod \"iptables-alerter-vps6j\" (UID: \"b7733c39-04f6-427d-bda9-9071b83f615c\") " pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002393    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-etc-kubernetes\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6-hosts-file\") pod \"node-resolver-ztrxp\" (UID: \"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6\") " pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-daemon-config\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002444    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-cni-dir\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2b45f30-896c-434f-b5a5-029893e1fd1b-konnectivity-ca\") pod \"konnectivity-agent-jw9j8\" (UID: \"c2b45f30-896c-434f-b5a5-029893e1fd1b\") " pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdgft\" (UniqueName: \"kubernetes.io/projected/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-kube-api-access-rdgft\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6-tmp-dir\") pod \"node-resolver-ztrxp\" (UID: \"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6\") " pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn82w\" (UniqueName: \"kubernetes.io/projected/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-kube-api-access-dn82w\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-kubelet\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.004326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-var-lib-openvswitch\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/375948f8-eecf-4ce7-a0ab-195b78a6d99f-ovnkube-config\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-etc-kubernetes\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-sysctl-d\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002776    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-host\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-var-lib-cni-multus\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6-hosts-file\") pod \"node-resolver-ztrxp\" (UID: \"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6\") " pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-cnibin\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/175a2b7e-f1c9-4a5c-bc4c-75f73cc79808-host\") pod \"node-ca-rxfgd\" (UID: \"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808\") " pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/175a2b7e-f1c9-4a5c-bc4c-75f73cc79808-serviceca\") pod \"node-ca-rxfgd\" (UID: \"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808\") " pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee80103b-11e9-42f0-a4d2-345e094101a6-cni-binary-copy\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-device-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-run\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-run\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-sysctl-d\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-host\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-var-lib-cni-multus\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-cnibin\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.005032 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/175a2b7e-f1c9-4a5c-bc4c-75f73cc79808-host\") pod \"node-ca-rxfgd\" (UID: \"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808\") " pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002859    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-multus-cni-dir\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/375948f8-eecf-4ce7-a0ab-195b78a6d99f-env-overrides\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c2b45f30-896c-434f-b5a5-029893e1fd1b-konnectivity-ca\") pod \"konnectivity-agent-jw9j8\" (UID: \"c2b45f30-896c-434f-b5a5-029893e1fd1b\") " pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-device-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-host-kubelet\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.002248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee80103b-11e9-42f0-a4d2-345e094101a6-host-run-multus-certs\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/375948f8-eecf-4ce7-a0ab-195b78a6d99f-var-lib-openvswitch\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-sysctl-conf\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/175a2b7e-f1c9-4a5c-bc4c-75f73cc79808-serviceca\") pod \"node-ca-rxfgd\" (UID: \"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808\") " pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6-tmp-dir\") pod \"node-resolver-ztrxp\" (UID: \"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6\") " pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003773    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/375948f8-eecf-4ce7-a0ab-195b78a6d99f-ovnkube-config\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7733c39-04f6-427d-bda9-9071b83f615c-iptables-alerter-script\") pod \"iptables-alerter-vps6j\" (UID: \"b7733c39-04f6-427d-bda9-9071b83f615c\") " pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.003893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.003997    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.004065    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs podName:7bb2809e-c82b-4a2b-86e6-dc341e864dfd nodeName:}" failed. No retries permitted until 2026-09-09 19:54:53.504047064 +0000 UTC m=+2.092700702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs") pod "network-metrics-daemon-8r42k" (UID: "7bb2809e-c82b-4a2b-86e6-dc341e864dfd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:53.005555 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.004292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b7733c39-04f6-427d-bda9-9071b83f615c-iptables-alerter-script\") pod \"iptables-alerter-vps6j\" (UID: \"b7733c39-04f6-427d-bda9-9071b83f615c\") " pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:53.006057 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.005084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4c2aac3-9593-4d44-9e55-92a0940bd662-etc-tuned\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.006057 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.005089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/375948f8-eecf-4ce7-a0ab-195b78a6d99f-ovn-node-metrics-cert\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.006057 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.005233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c2b45f30-896c-434f-b5a5-029893e1fd1b-agent-certs\") pod \"konnectivity-agent-jw9j8\" (UID: \"c2b45f30-896c-434f-b5a5-029893e1fd1b\") " pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:54:53.006057 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.005554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4c2aac3-9593-4d44-9e55-92a0940bd662-tmp\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.012875 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.012852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lgh9\" (UniqueName: \"kubernetes.io/projected/175a2b7e-f1c9-4a5c-bc4c-75f73cc79808-kube-api-access-9lgh9\") pod \"node-ca-rxfgd\" (UID: \"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808\") " pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:53.014458 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.014418    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qwn\" (UniqueName: \"kubernetes.io/projected/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-kube-api-access-f4qwn\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:53.016268 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.016244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdgft\" (UniqueName: \"kubernetes.io/projected/43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3-kube-api-access-rdgft\") pod \"aws-ebs-csi-driver-node-4nv8l\" (UID: \"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.016630 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.016610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn82w\" (UniqueName: \"kubernetes.io/projected/37cd81c0-a0cd-48ee-a49b-5496fedaf6d8-kube-api-access-dn82w\") pod \"multus-additional-cni-plugins-6b6dl\" (UID: \"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8\") " pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.019193 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.016683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrn6x\" (UniqueName: \"kubernetes.io/projected/b7733c39-04f6-427d-bda9-9071b83f615c-kube-api-access-vrn6x\") pod \"iptables-alerter-vps6j\" (UID: \"b7733c39-04f6-427d-bda9-9071b83f615c\") " pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:53.019193 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.017326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmrl\" (UniqueName: \"kubernetes.io/projected/d4c2aac3-9593-4d44-9e55-92a0940bd662-kube-api-access-jcmrl\") pod \"tuned-rj4vk\" (UID: \"d4c2aac3-9593-4d44-9e55-92a0940bd662\") " pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.019193 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.017401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr88b\" (UniqueName: \"kubernetes.io/projected/cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6-kube-api-access-wr88b\") pod \"node-resolver-ztrxp\" (UID: \"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6\") " pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:53.019193 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.017435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4zn\" (UniqueName: \"kubernetes.io/projected/ee80103b-11e9-42f0-a4d2-345e094101a6-kube-api-access-2q4zn\") pod \"multus-k6jj7\" (UID: \"ee80103b-11e9-42f0-a4d2-345e094101a6\") " pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.019763 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.019744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9ts\" (UniqueName: \"kubernetes.io/projected/375948f8-eecf-4ce7-a0ab-195b78a6d99f-kube-api-access-5m9ts\") pod \"ovnkube-node-ccdvx\" (UID: \"375948f8-eecf-4ce7-a0ab-195b78a6d99f\") " pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.029754 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.029736    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:53.050902 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.050867    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181a392e75f0abd87761766443396815.slice/crio-bf94999dc1afb2efc482faa6d502d5a86a439e105ea3e3947e20af81e40eb177 WatchSource:0}: Error finding container bf94999dc1afb2efc482faa6d502d5a86a439e105ea3e3947e20af81e40eb177: Status 404 returned error can't find the container with id bf94999dc1afb2efc482faa6d502d5a86a439e105ea3e3947e20af81e40eb177
Sep 09 19:54:53.058299 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.058283    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 19:54:53.068168 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.068146    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7836c45ac94a1983d60a015a961f0779.slice/crio-dcdb746c32d19758409d5f419d3902a27f3fd3f37d48ff16204d2ddb28644689 WatchSource:0}: Error finding container dcdb746c32d19758409d5f419d3902a27f3fd3f37d48ff16204d2ddb28644689: Status 404 returned error can't find the container with id dcdb746c32d19758409d5f419d3902a27f3fd3f37d48ff16204d2ddb28644689
Sep 09 19:54:53.113433 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.113370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vps6j"
Sep 09 19:54:53.121907 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.121887    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7733c39_04f6_427d_bda9_9071b83f615c.slice/crio-93a97c1a879ad004a738861b903e97cf207dcdd657dcdca9e1172ae5069de2e0 WatchSource:0}: Error finding container 93a97c1a879ad004a738861b903e97cf207dcdd657dcdca9e1172ae5069de2e0: Status 404 returned error can't find the container with id 93a97c1a879ad004a738861b903e97cf207dcdd657dcdca9e1172ae5069de2e0
Sep 09 19:54:53.139051 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.139029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6b6dl"
Sep 09 19:54:53.146621 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.146602    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cd81c0_a0cd_48ee_a49b_5496fedaf6d8.slice/crio-8796f621f55ebda18422b5f293a4d0c548e45ff29b020c9dd21559104223918c WatchSource:0}: Error finding container 8796f621f55ebda18422b5f293a4d0c548e45ff29b020c9dd21559104223918c: Status 404 returned error can't find the container with id 8796f621f55ebda18422b5f293a4d0c548e45ff29b020c9dd21559104223918c
Sep 09 19:54:53.160761 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.160736    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:54:53.164644 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.164624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rxfgd"
Sep 09 19:54:53.169747 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.169718    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375948f8_eecf_4ce7_a0ab_195b78a6d99f.slice/crio-4f800367bb30c72434ea2f1b22e727507cec8446814da9a65f60682fa3605891 WatchSource:0}: Error finding container 4f800367bb30c72434ea2f1b22e727507cec8446814da9a65f60682fa3605891: Status 404 returned error can't find the container with id 4f800367bb30c72434ea2f1b22e727507cec8446814da9a65f60682fa3605891
Sep 09 19:54:53.174794 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.174770    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175a2b7e_f1c9_4a5c_bc4c_75f73cc79808.slice/crio-2b6524d12470fe74b387b24a4c8c4aa83226d0a979156b2b27a3be2b125ba87e WatchSource:0}: Error finding container 2b6524d12470fe74b387b24a4c8c4aa83226d0a979156b2b27a3be2b125ba87e: Status 404 returned error can't find the container with id 2b6524d12470fe74b387b24a4c8c4aa83226d0a979156b2b27a3be2b125ba87e
Sep 09 19:54:53.178787 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.178772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l"
Sep 09 19:54:53.187349 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.187331    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ca0c32_0b8c_4f0a_a9d2_01bf2165bcd3.slice/crio-2cc4647e8905405ba99bed3d0cfbb80dd0869e4f6ee07c633a29c1d57624100c WatchSource:0}: Error finding container 2cc4647e8905405ba99bed3d0cfbb80dd0869e4f6ee07c633a29c1d57624100c: Status 404 returned error can't find the container with id 2cc4647e8905405ba99bed3d0cfbb80dd0869e4f6ee07c633a29c1d57624100c
Sep 09 19:54:53.197403 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.197384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rj4vk"
Sep 09 19:54:53.204645 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.204627    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c2aac3_9593_4d44_9e55_92a0940bd662.slice/crio-2248287e8d4f315d1b5d219768146a940245e8db8007f15bdb20c000d0daade2 WatchSource:0}: Error finding container 2248287e8d4f315d1b5d219768146a940245e8db8007f15bdb20c000d0daade2: Status 404 returned error can't find the container with id 2248287e8d4f315d1b5d219768146a940245e8db8007f15bdb20c000d0daade2
Sep 09 19:54:53.212319 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.212289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ztrxp"
Sep 09 19:54:53.220162 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.220142    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc50338f_4fdf_41c7_a40e_19d0e3ebb9c6.slice/crio-9124b015d5accf4634c9fb9af511940de835c9d406ea9c2f13ae76afebb0ee92 WatchSource:0}: Error finding container 9124b015d5accf4634c9fb9af511940de835c9d406ea9c2f13ae76afebb0ee92: Status 404 returned error can't find the container with id 9124b015d5accf4634c9fb9af511940de835c9d406ea9c2f13ae76afebb0ee92
Sep 09 19:54:53.220694 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.220673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-k6jj7"
Sep 09 19:54:53.224395 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.224378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:54:53.228211 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.228192    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee80103b_11e9_42f0_a4d2_345e094101a6.slice/crio-d85f9be967cd7fa6e527fa3db0b139764a98a5cc2e5154873689763bc548a52e WatchSource:0}: Error finding container d85f9be967cd7fa6e527fa3db0b139764a98a5cc2e5154873689763bc548a52e: Status 404 returned error can't find the container with id d85f9be967cd7fa6e527fa3db0b139764a98a5cc2e5154873689763bc548a52e
Sep 09 19:54:53.232277 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:54:53.232259    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b45f30_896c_434f_b5a5_029893e1fd1b.slice/crio-71bc9bfeee840607fcef85c90a5d8d7d8e958507a065dfbf66c58739dc0f8c8b WatchSource:0}: Error finding container 71bc9bfeee840607fcef85c90a5d8d7d8e958507a065dfbf66c58739dc0f8c8b: Status 404 returned error can't find the container with id 71bc9bfeee840607fcef85c90a5d8d7d8e958507a065dfbf66c58739dc0f8c8b
Sep 09 19:54:53.233894 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.233879    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-jw9j8_kube-system(c2b45f30-896c-434f-b5a5-029893e1fd1b): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 09 19:54:53.233944 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.233909    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-jw9j8" podUID="c2b45f30-896c-434f-b5a5-029893e1fd1b"
Sep 09 19:54:53.406541 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.406458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:54:53.406681 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.406612    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:54:53.406681 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.406629    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:54:53.406681 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.406637    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x8vd5 for pod openshift-network-diagnostics/network-check-target-t7mhh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:53.406784 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.406695    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5 podName:bc10e010-ba02-48e7-acec-96938127af83 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:54.406679285 +0000 UTC m=+2.995332939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x8vd5" (UniqueName: "kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5") pod "network-check-target-t7mhh" (UID: "bc10e010-ba02-48e7-acec-96938127af83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:53.507570 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.507534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:53.507570 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.507572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:53.507754 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.507663    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:53.507754 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.507675    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:53.507754 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.507713    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret podName:34db695b-0f6e-43c2-b92f-65214c6f28e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:54.5077004 +0000 UTC m=+3.096354034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret") pod "global-pull-secret-syncer-2458j" (UID: "34db695b-0f6e-43c2-b92f-65214c6f28e4") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:53.507754 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.507727    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs podName:7bb2809e-c82b-4a2b-86e6-dc341e864dfd nodeName:}" failed. No retries permitted until 2026-09-09 19:54:54.507720813 +0000 UTC m=+3.096374449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs") pod "network-metrics-daemon-8r42k" (UID: "7bb2809e-c82b-4a2b-86e6-dc341e864dfd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:53.659707 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.659629    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:53.840010 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.839950    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:49:52 +0000 UTC" deadline="2028-02-07 23:41:08.098171998 +0000 UTC"
Sep 09 19:54:53.840010 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.839996    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12387h46m14.258180221s"
Sep 09 19:54:53.898751 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.898661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b6dl" event={"ID":"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8","Type":"ContainerStarted","Data":"8796f621f55ebda18422b5f293a4d0c548e45ff29b020c9dd21559104223918c"}
Sep 09 19:54:53.900363 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.900330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vps6j" event={"ID":"b7733c39-04f6-427d-bda9-9071b83f615c","Type":"ContainerStarted","Data":"93a97c1a879ad004a738861b903e97cf207dcdd657dcdca9e1172ae5069de2e0"}
Sep 09 19:54:53.902546 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.902525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k6jj7" event={"ID":"ee80103b-11e9-42f0-a4d2-345e094101a6","Type":"ContainerStarted","Data":"d85f9be967cd7fa6e527fa3db0b139764a98a5cc2e5154873689763bc548a52e"}
Sep 09 19:54:53.904909 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.904852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rj4vk" event={"ID":"d4c2aac3-9593-4d44-9e55-92a0940bd662","Type":"ContainerStarted","Data":"2248287e8d4f315d1b5d219768146a940245e8db8007f15bdb20c000d0daade2"}
Sep 09 19:54:53.907271 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.907219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l" event={"ID":"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3","Type":"ContainerStarted","Data":"2cc4647e8905405ba99bed3d0cfbb80dd0869e4f6ee07c633a29c1d57624100c"}
Sep 09 19:54:53.910388 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.910330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rxfgd" event={"ID":"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808","Type":"ContainerStarted","Data":"2b6524d12470fe74b387b24a4c8c4aa83226d0a979156b2b27a3be2b125ba87e"}
Sep 09 19:54:53.912812 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.912788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal" event={"ID":"7836c45ac94a1983d60a015a961f0779","Type":"ContainerStarted","Data":"dcdb746c32d19758409d5f419d3902a27f3fd3f37d48ff16204d2ddb28644689"}
Sep 09 19:54:53.914489 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.914461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal" event={"ID":"181a392e75f0abd87761766443396815","Type":"ContainerStarted","Data":"bf94999dc1afb2efc482faa6d502d5a86a439e105ea3e3947e20af81e40eb177"}
Sep 09 19:54:53.917350 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.917325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jw9j8" event={"ID":"c2b45f30-896c-434f-b5a5-029893e1fd1b","Type":"ContainerStarted","Data":"71bc9bfeee840607fcef85c90a5d8d7d8e958507a065dfbf66c58739dc0f8c8b"}
Sep 09 19:54:53.918941 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:53.918912    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9018c47455af82aa9aba488b18f1fb9824a505ad46539494602fe936ffc4b601\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-jw9j8" podUID="c2b45f30-896c-434f-b5a5-029893e1fd1b"
Sep 09 19:54:53.920288 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.920262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ztrxp" event={"ID":"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6","Type":"ContainerStarted","Data":"9124b015d5accf4634c9fb9af511940de835c9d406ea9c2f13ae76afebb0ee92"}
Sep 09 19:54:53.923420 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:53.923396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerStarted","Data":"4f800367bb30c72434ea2f1b22e727507cec8446814da9a65f60682fa3605891"}
Sep 09 19:54:54.414172 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.414128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:54:54.414449 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.414402    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:54:54.414449 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.414428    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:54:54.414449 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.414441    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x8vd5 for pod openshift-network-diagnostics/network-check-target-t7mhh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:54.414668 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.414504    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5 podName:bc10e010-ba02-48e7-acec-96938127af83 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:56.414485176 +0000 UTC m=+5.003138816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x8vd5" (UniqueName: "kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5") pod "network-check-target-t7mhh" (UID: "bc10e010-ba02-48e7-acec-96938127af83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:54.467821 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.467790    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:54.514776 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.514738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:54.514953 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.514793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:54.514953 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.514923    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:54.515069 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.514983    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret podName:34db695b-0f6e-43c2-b92f-65214c6f28e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:56.514965885 +0000 UTC m=+5.103619522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret") pod "global-pull-secret-syncer-2458j" (UID: "34db695b-0f6e-43c2-b92f-65214c6f28e4") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:54.515393 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.515373    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:54.515490 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.515437    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs podName:7bb2809e-c82b-4a2b-86e6-dc341e864dfd nodeName:}" failed. No retries permitted until 2026-09-09 19:54:56.515422252 +0000 UTC m=+5.104075887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs") pod "network-metrics-daemon-8r42k" (UID: "7bb2809e-c82b-4a2b-86e6-dc341e864dfd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:54.841070 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.841028    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:49:52 +0000 UTC" deadline="2028-06-08 09:10:15.773730134 +0000 UTC"
Sep 09 19:54:54.841070 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.841071    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15301h15m20.932663176s"
Sep 09 19:54:54.860968 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.860934    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:54.890894 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.890863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:54:54.891058 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.890982    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:54:54.891142 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.891061    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:54.891142 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.891131    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:54:54.894854 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:54.894823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:54.894998 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:54.894978    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:54:56.431725 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:56.431616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:54:56.432267 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.431775    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:54:56.432267 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.431794    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:54:56.432267 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.431806    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x8vd5 for pod openshift-network-diagnostics/network-check-target-t7mhh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:56.432267 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.431861    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5 podName:bc10e010-ba02-48e7-acec-96938127af83 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:00.431843627 +0000 UTC m=+9.020497265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x8vd5" (UniqueName: "kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5") pod "network-check-target-t7mhh" (UID: "bc10e010-ba02-48e7-acec-96938127af83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:56.532838 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:56.532799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:56.533046 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:56.532858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:56.533046 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.533015    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:56.533181 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.533080    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret podName:34db695b-0f6e-43c2-b92f-65214c6f28e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:00.533061604 +0000 UTC m=+9.121715262 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret") pod "global-pull-secret-syncer-2458j" (UID: "34db695b-0f6e-43c2-b92f-65214c6f28e4") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:56.533181 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.533109    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:56.533181 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.533175    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs podName:7bb2809e-c82b-4a2b-86e6-dc341e864dfd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:00.533157299 +0000 UTC m=+9.121810948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs") pod "network-metrics-daemon-8r42k" (UID: "7bb2809e-c82b-4a2b-86e6-dc341e864dfd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:56.890557 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:56.890523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:54:56.890755 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.890660    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:54:56.891340 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:56.891049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:56.891340 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.891151    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:54:56.891340 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:56.891201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:56.891340 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:56.891278    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:54:58.890670 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:58.890213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:54:58.890670 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:58.890369    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:54:58.890670 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:58.890392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:54:58.890670 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:58.890474    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:54:58.890670 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:54:58.890519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:54:58.890670 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:54:58.890596    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:00.464365 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:00.464322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:00.464875 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.464460    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:00.464875 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.464491    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:00.464875 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.464507    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x8vd5 for pod openshift-network-diagnostics/network-check-target-t7mhh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:00.464875 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.464572    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5 podName:bc10e010-ba02-48e7-acec-96938127af83 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:08.46455283 +0000 UTC m=+17.053206484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x8vd5" (UniqueName: "kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5") pod "network-check-target-t7mhh" (UID: "bc10e010-ba02-48e7-acec-96938127af83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:00.565186 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:00.565141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:00.565381 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:00.565197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:00.565381 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.565329    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:00.565381 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.565341    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:00.565540 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.565403    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret podName:34db695b-0f6e-43c2-b92f-65214c6f28e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:08.565386111 +0000 UTC m=+17.154039750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret") pod "global-pull-secret-syncer-2458j" (UID: "34db695b-0f6e-43c2-b92f-65214c6f28e4") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:00.565540 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.565423    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs podName:7bb2809e-c82b-4a2b-86e6-dc341e864dfd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:08.565412674 +0000 UTC m=+17.154066309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs") pod "network-metrics-daemon-8r42k" (UID: "7bb2809e-c82b-4a2b-86e6-dc341e864dfd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:00.890723 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:00.890678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:00.890883 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:00.890770    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:00.890883 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:00.890794    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:00.891012 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.890936    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:00.891222 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.891178    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:00.891377 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:00.891279    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:02.891041 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:02.891001    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:02.891516 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:02.891137    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:02.891516 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:02.891234    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:02.891516 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:02.891136    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:02.891516 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:02.891274    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:02.891516 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:02.891394    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:04.890899 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:04.890843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:04.890899 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:04.890876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:04.891491 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:04.890865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:04.891491 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:04.890979    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:04.891491 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:04.891070    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:04.891491 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:04.891177    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:06.890585 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:06.890554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:06.890949 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:06.890554    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:06.890949 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:06.890664    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:06.890949 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:06.890553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:06.890949 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:06.890727    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:06.890949 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:06.890823    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:08.525264 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:08.525138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:08.525750 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.525287    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:08.525750 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.525326    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:08.525750 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.525341    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x8vd5 for pod openshift-network-diagnostics/network-check-target-t7mhh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:08.525750 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.525405    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5 podName:bc10e010-ba02-48e7-acec-96938127af83 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:24.525385353 +0000 UTC m=+33.114039011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x8vd5" (UniqueName: "kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5") pod "network-check-target-t7mhh" (UID: "bc10e010-ba02-48e7-acec-96938127af83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:08.626212 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:08.626180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:08.626398 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:08.626226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:08.626398 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.626351    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:08.626398 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.626379    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:08.626540 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.626409    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret podName:34db695b-0f6e-43c2-b92f-65214c6f28e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:24.626391591 +0000 UTC m=+33.215045226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret") pod "global-pull-secret-syncer-2458j" (UID: "34db695b-0f6e-43c2-b92f-65214c6f28e4") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:08.626540 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.626446    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs podName:7bb2809e-c82b-4a2b-86e6-dc341e864dfd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:24.626418601 +0000 UTC m=+33.215072253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs") pod "network-metrics-daemon-8r42k" (UID: "7bb2809e-c82b-4a2b-86e6-dc341e864dfd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:08.891037 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:08.891002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:08.891238 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:08.891013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:08.891238 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:08.891128    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:08.891238 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.891123    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:08.891417 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.891240    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:08.891417 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:08.891326    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:10.890429 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:10.890392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:10.890990 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:10.890463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:10.890990 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:10.890497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:10.890990 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:10.890589    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:10.890990 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:10.890648    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:10.890990 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:10.890757    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:11.971681 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:11.971419    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 19:55:11.973270 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:11.972968    2578 generic.go:356] "Generic (PLEG): container finished" podID="375948f8-eecf-4ce7-a0ab-195b78a6d99f" containerID="e230afe7c37e50250916d27c8dfcf392c24db2ea115234bc092c4ce4c2df15f5" exitCode=1
Sep 09 19:55:11.973270 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:11.973061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerStarted","Data":"e073f00d882a5d42bdb18b625d63f2fdba50a5710331e755c2ccd3994da0a3e3"}
Sep 09 19:55:11.973270 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:11.973140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerDied","Data":"e230afe7c37e50250916d27c8dfcf392c24db2ea115234bc092c4ce4c2df15f5"}
Sep 09 19:55:11.973270 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:11.973195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerStarted","Data":"dce6596b64af1577041eb222b9eb2043b51e18c2256789736f13900d84e41f3a"}
Sep 09 19:55:11.981911 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:11.981850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-k6jj7" event={"ID":"ee80103b-11e9-42f0-a4d2-345e094101a6","Type":"ContainerStarted","Data":"f2905c580f98243c104c7c1e002e661702083bf74317186e77439d3755dd7684"}
Sep 09 19:55:11.989765 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:11.989354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rj4vk" event={"ID":"d4c2aac3-9593-4d44-9e55-92a0940bd662","Type":"ContainerStarted","Data":"1ca2e0ee3b4af523ff40d17594ed4b3eab88fb48cbcba61f2ae9d614347178a2"}
Sep 09 19:55:11.997553 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:11.997484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal" event={"ID":"181a392e75f0abd87761766443396815","Type":"ContainerStarted","Data":"801929e15531bf20caa1b886f244cebc1718f283634bc46939dbb29efa627d8e"}
Sep 09 19:55:12.020693 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:12.019747    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-k6jj7" podStartSLOduration=1.408675694 podStartE2EDuration="20.019726069s" totalImagesPullingTime="18.611050375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:12.000524173 +0000 UTC m=+20.589177864" watchObservedRunningTime="2026-09-09 19:55:12.019726069 +0000 UTC m=+20.608379707"
Sep 09 19:55:12.035467 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:12.035422    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rj4vk" podStartSLOduration=1.934583132 podStartE2EDuration="20.035400577s" totalImagesPullingTime="18.100817445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:12.020607993 +0000 UTC m=+20.609261628" watchObservedRunningTime="2026-09-09 19:55:12.035400577 +0000 UTC m=+20.624054234"
Sep 09 19:55:12.036507 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:12.036378    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-141.ec2.internal" podStartSLOduration=20.03635934 podStartE2EDuration="20.03635934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:12.035467308 +0000 UTC m=+20.624120967" watchObservedRunningTime="2026-09-09 19:55:12.03635934 +0000 UTC m=+20.625012998"
Sep 09 19:55:12.890849 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:12.890820    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:12.890849 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:12.890846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:12.890976 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:12.890931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:12.890976 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:12.890932    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:12.891068 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:12.890996    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:12.891120 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:12.891083    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:13.001070 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.000979    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cd81c0-a0cd-48ee-a49b-5496fedaf6d8" containerID="72d4fd6f26e78c423d8910a368f7596886a88cb26e45aa7ecb118f3fa1d252a4" exitCode=0
Sep 09 19:55:13.001070 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.001051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b6dl" event={"ID":"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8","Type":"ContainerDied","Data":"72d4fd6f26e78c423d8910a368f7596886a88cb26e45aa7ecb118f3fa1d252a4"}
Sep 09 19:55:13.002589 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.002561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vps6j" event={"ID":"b7733c39-04f6-427d-bda9-9071b83f615c","Type":"ContainerStarted","Data":"4b1024770cc864c26582971b8b24a964975624f8b65b6e27905a3643878fc2ee"}
Sep 09 19:55:13.004104 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.004059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l" event={"ID":"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3","Type":"ContainerStarted","Data":"008b6cc16ba2fec526bac439185992e111b4159d0caab2839c26362c2039be0a"}
Sep 09 19:55:13.005461 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.005442    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rxfgd" event={"ID":"175a2b7e-f1c9-4a5c-bc4c-75f73cc79808","Type":"ContainerStarted","Data":"3725249dcc1a74c1660d8e11f09d94fc3d709cbe5b166ed79fc910c6e6a3abc6"}
Sep 09 19:55:13.007018 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.006996    2578 generic.go:356] "Generic (PLEG): container finished" podID="7836c45ac94a1983d60a015a961f0779" containerID="2722b93616228cfb98c2634ec5e3b0db7f49417bb13a16403148a63b4c470ac4" exitCode=0
Sep 09 19:55:13.007127 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.007056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal" event={"ID":"7836c45ac94a1983d60a015a961f0779","Type":"ContainerDied","Data":"2722b93616228cfb98c2634ec5e3b0db7f49417bb13a16403148a63b4c470ac4"}
Sep 09 19:55:13.008585 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.008560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jw9j8" event={"ID":"c2b45f30-896c-434f-b5a5-029893e1fd1b","Type":"ContainerStarted","Data":"4601a21d00cefb044a7a4a2a765c72698a32a5671ca1df64da6545560d34fbf0"}
Sep 09 19:55:13.009947 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.009926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ztrxp" event={"ID":"cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6","Type":"ContainerStarted","Data":"dd440bf441415ff35b27889d157ae5a9d5396539360f15956557dda2f27a6a1d"}
Sep 09 19:55:13.012587 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.012570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 19:55:13.012918 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.012893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerStarted","Data":"73eb4d3b48996fe713326161b28178a6f6955362fc857aef1970d4598ed892bc"}
Sep 09 19:55:13.012971 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.012947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerStarted","Data":"e2c26592c8df0d1f468b17199c2a8807e1ce93740f06bf21f917ea6ad817d2e7"}
Sep 09 19:55:13.013005 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.012978    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerStarted","Data":"e0dc9b27bc24b7327cfe341650aadc2b4a36dcfd43eda68072962c129432e83b"}
Sep 09 19:55:13.043153 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.042927    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ztrxp" podStartSLOduration=2.9572020930000003 podStartE2EDuration="21.04290761s" totalImagesPullingTime="18.085705517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:13.042246521 +0000 UTC m=+21.630900179" watchObservedRunningTime="2026-09-09 19:55:13.04290761 +0000 UTC m=+21.631561272"
Sep 09 19:55:13.070987 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.070949    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vps6j" podStartSLOduration=3.888094117 podStartE2EDuration="22.07093402s" totalImagesPullingTime="18.182839903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:13.070854994 +0000 UTC m=+21.659508652" watchObservedRunningTime="2026-09-09 19:55:13.07093402 +0000 UTC m=+21.659587716"
Sep 09 19:55:13.084036 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.083995    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jw9j8" podStartSLOduration=1.539635978 podStartE2EDuration="21.083981303s" totalImagesPullingTime="19.544345325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-09 19:55:13.083851236 +0000 UTC m=+21.672504894" watchObservedRunningTime="2026-09-09 19:55:13.083981303 +0000 UTC m=+21.672634959"
Sep 09 19:55:13.098231 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:13.098190    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rxfgd" podStartSLOduration=2.967518911 podStartE2EDuration="21.098177489s" totalImagesPullingTime="18.130658578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:13.097972251 +0000 UTC m=+21.686625909" watchObservedRunningTime="2026-09-09 19:55:13.098177489 +0000 UTC m=+21.686831145"
Sep 09 19:55:14.017903 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:14.017860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal" event={"ID":"7836c45ac94a1983d60a015a961f0779","Type":"ContainerStarted","Data":"474cf5d396623166f484baea6bee1c1c963f9ce5d2ec4987637ef45dd105a9be"}
Sep 09 19:55:14.045222 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:14.045171    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-141.ec2.internal" podStartSLOduration=22.045155603 podStartE2EDuration="22.045155603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:14.045102374 +0000 UTC m=+22.633756031" watchObservedRunningTime="2026-09-09 19:55:14.045155603 +0000 UTC m=+22.633809261"
Sep 09 19:55:14.060347 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:14.060322    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 19:55:14.869964 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:14.869868    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T19:55:14.060343063Z","UUID":"870656b9-e864-4478-b6f7-e3526814925f","Handler":null,"Name":"","Endpoint":""}
Sep 09 19:55:14.871388 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:14.871361    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 19:55:14.871388 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:14.871392    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 19:55:14.890271 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:14.890241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:14.890271 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:14.890273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:14.890446 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:14.890241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:14.890446 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:14.890364    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:14.890516 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:14.890492    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:14.890583 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:14.890566    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:15.023856 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:15.023825    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 19:55:15.024349 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:15.024300    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerStarted","Data":"44bf51fcd94a249884a3ec20b26cdc346828ce3281cd12a12cbe36c3ab71aa9d"}
Sep 09 19:55:15.026692 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:15.026666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l" event={"ID":"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3","Type":"ContainerStarted","Data":"457821f12adf323a2de32946966992d6ab8e728e47efcc6a6595f76e2ca55aed"}
Sep 09 19:55:15.026798 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:15.026723    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l" event={"ID":"43ca0c32-0b8c-4f0a-a9d2-01bf2165bcd3","Type":"ContainerStarted","Data":"4577d416ce4857488f0f7dae87bf747a7e4d21f933e5446051b0883d902201f1"}
Sep 09 19:55:15.044753 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:15.044716    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4nv8l" podStartSLOduration=2.291401134 podStartE2EDuration="23.044701982s" totalImagesPullingTime="20.753300848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:15.044210594 +0000 UTC m=+23.632864253" watchObservedRunningTime="2026-09-09 19:55:15.044701982 +0000 UTC m=+23.633355639"
Sep 09 19:55:16.777058 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:16.776881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:55:16.777541 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:16.777489    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:55:16.890483 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:16.890452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:16.890680 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:16.890489    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:16.890680 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:16.890452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:16.890680 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:16.890589    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:16.890680 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:16.890667    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:16.890878 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:16.890730    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:17.035540 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:17.035463    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 19:55:17.035941 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:17.035908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerStarted","Data":"2a9818fb17142f5baf860bb6edd2c2ce0429370a84cb8b2a13c604f95fb96539"}
Sep 09 19:55:17.036060 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:17.036018    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:55:17.036416 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:17.036396    2578 scope.go:117] "RemoveContainer" containerID="e230afe7c37e50250916d27c8dfcf392c24db2ea115234bc092c4ce4c2df15f5"
Sep 09 19:55:17.036673 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:17.036656    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jw9j8"
Sep 09 19:55:18.291565 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:18.291525    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2458j"]
Sep 09 19:55:18.292291 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:18.291658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:18.292291 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:18.291759    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:18.295515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:18.295321    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8r42k"]
Sep 09 19:55:18.295515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:18.295439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:18.295701 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:18.295535    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:18.296199 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:18.296174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7mhh"]
Sep 09 19:55:18.296335 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:18.296264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:18.296402 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:18.296367    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:19.044124 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.044066    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 19:55:19.044459 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.044436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" event={"ID":"375948f8-eecf-4ce7-a0ab-195b78a6d99f","Type":"ContainerStarted","Data":"7c76c6b254c0d5033f6c530e1c027f8f4ec6f54e8e2c5319761ec32c877750da"}
Sep 09 19:55:19.044716 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.044692    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:55:19.045092 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.044857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:55:19.045092 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.044946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:55:19.046457 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.046432    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cd81c0-a0cd-48ee-a49b-5496fedaf6d8" containerID="953e59c8c7d980282f6991675313c7785a2a386e6136bd3e251231d28d0909b0" exitCode=0
Sep 09 19:55:19.046592 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.046518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b6dl" event={"ID":"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8","Type":"ContainerDied","Data":"953e59c8c7d980282f6991675313c7785a2a386e6136bd3e251231d28d0909b0"}
Sep 09 19:55:19.060179 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.060158    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:55:19.060302 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.060287    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:55:19.076853 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.076763    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" podStartSLOduration=8.924354086 podStartE2EDuration="27.07674918s" totalImagesPullingTime="18.152395094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:19.075592812 +0000 UTC m=+27.664246468" watchObservedRunningTime="2026-09-09 19:55:19.07674918 +0000 UTC m=+27.665402834"
Sep 09 19:55:19.891016 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.890980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:19.891589 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.891026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:19.891589 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:19.891037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:19.891589 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:19.891106    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:19.891589 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:19.891169    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:19.891589 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:19.891233    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:21.053986 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:21.053812    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cd81c0-a0cd-48ee-a49b-5496fedaf6d8" containerID="f22f86171be6490067ea0caee0f332d03655dc4d2c8ffc92fe1261e8fdd25c8d" exitCode=0
Sep 09 19:55:21.054391 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:21.053894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b6dl" event={"ID":"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8","Type":"ContainerDied","Data":"f22f86171be6490067ea0caee0f332d03655dc4d2c8ffc92fe1261e8fdd25c8d"}
Sep 09 19:55:21.070261 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:21.070229    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx" podUID="375948f8-eecf-4ce7-a0ab-195b78a6d99f" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 09 19:55:21.891137 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:21.891109    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:21.891252 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:21.891190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:21.891334 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:21.891192    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:21.891334 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:21.891328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:21.891432 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:21.891331    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:21.891432 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:21.891416    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:22.058823 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:22.058789    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cd81c0-a0cd-48ee-a49b-5496fedaf6d8" containerID="9173e1a70674a091404eaed7e166b9034cc5dc6d305cb2b6171f627fdf0bfc08" exitCode=0
Sep 09 19:55:22.059199 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:22.058853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b6dl" event={"ID":"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8","Type":"ContainerDied","Data":"9173e1a70674a091404eaed7e166b9034cc5dc6d305cb2b6171f627fdf0bfc08"}
Sep 09 19:55:23.890915 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:23.890883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:23.890915 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:23.890874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:23.891443 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:23.890927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:23.891443 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:23.891023    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8r42k" podUID="7bb2809e-c82b-4a2b-86e6-dc341e864dfd"
Sep 09 19:55:23.891539 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:23.891482    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2458j" podUID="34db695b-0f6e-43c2-b92f-65214c6f28e4"
Sep 09 19:55:23.891588 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:23.891120    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7mhh" podUID="bc10e010-ba02-48e7-acec-96938127af83"
Sep 09 19:55:24.535339 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.535287    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:24.535618 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.535499    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:24.535618 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.535527    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:24.535618 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.535540    2578 projected.go:196] Error preparing data for projected volume kube-api-access-x8vd5 for pod openshift-network-diagnostics/network-check-target-t7mhh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:24.535782 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.535663    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5 podName:bc10e010-ba02-48e7-acec-96938127af83 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.535619081 +0000 UTC m=+65.124272736 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x8vd5" (UniqueName: "kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5") pod "network-check-target-t7mhh" (UID: "bc10e010-ba02-48e7-acec-96938127af83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:24.636055 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.636020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:24.636055 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.636061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:24.636299 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.636181    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:24.636299 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.636184    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:24.636299 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.636243    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret podName:34db695b-0f6e-43c2-b92f-65214c6f28e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.636224679 +0000 UTC m=+65.224878339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret") pod "global-pull-secret-syncer-2458j" (UID: "34db695b-0f6e-43c2-b92f-65214c6f28e4") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:24.636299 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.636260    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs podName:7bb2809e-c82b-4a2b-86e6-dc341e864dfd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.636251122 +0000 UTC m=+65.224904756 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs") pod "network-metrics-daemon-8r42k" (UID: "7bb2809e-c82b-4a2b-86e6-dc341e864dfd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:24.670739 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.670710    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-141.ec2.internal" event="NodeReady"
Sep 09 19:55:24.670884 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.670864    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 19:55:24.712203 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.712176    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55bc687dd7-kf6hm"]
Sep 09 19:55:24.735611 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.735585    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-82kmf"]
Sep 09 19:55:24.735756 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.735743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.738805 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.738782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xdvvq\""
Sep 09 19:55:24.739059 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.739041    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 19:55:24.739151 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.739044    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 19:55:24.739151 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.739075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 19:55:24.745166 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.745117    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 19:55:24.751465 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.751446    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5x4l5"]
Sep 09 19:55:24.751566 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.751553    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.753812 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.753794    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5xtg\""
Sep 09 19:55:24.753883 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.753837    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 19:55:24.755173 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.755155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 19:55:24.772924 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.772891    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bc687dd7-kf6hm"]
Sep 09 19:55:24.772924 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.772929    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5x4l5"]
Sep 09 19:55:24.773124 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.772952    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-82kmf"]
Sep 09 19:55:24.773124 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.773028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5x4l5"
Sep 09 19:55:24.775781 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.775760    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 19:55:24.775937 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.775808    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 19:55:24.775937 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.775859    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gmppz\""
Sep 09 19:55:24.775937 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.775912    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 19:55:24.837197 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-bound-sa-token\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.837392 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.837392 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kqs\" (UniqueName: \"kubernetes.io/projected/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-kube-api-access-z4kqs\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.837392 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-image-registry-private-configuration\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.837392 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgjpf\" (UniqueName: \"kubernetes.io/projected/65266674-4ef3-4abf-b20c-3c06447af305-kube-api-access-pgjpf\") pod \"ingress-canary-5x4l5\" (UID: \"65266674-4ef3-4abf-b20c-3c06447af305\") " pod="openshift-ingress-canary/ingress-canary-5x4l5"
Sep 09 19:55:24.837392 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-tmp-dir\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.837640 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.837640 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-ca-trust-extracted\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.837640 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-trusted-ca\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.837640 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2zrj\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-kube-api-access-p2zrj\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.837640 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-certificates\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.838062 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.837630    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-installation-pull-secrets\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.838142 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.838099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-config-volume\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.838193 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.838140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65266674-4ef3-4abf-b20c-3c06447af305-cert\") pod \"ingress-canary-5x4l5\" (UID: \"65266674-4ef3-4abf-b20c-3c06447af305\") " pod="openshift-ingress-canary/ingress-canary-5x4l5"
Sep 09 19:55:24.901092 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.901015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:24.939088 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-certificates\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.939088 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939087    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-installation-pull-secrets\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.939298 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-config-volume\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.939298 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65266674-4ef3-4abf-b20c-3c06447af305-cert\") pod \"ingress-canary-5x4l5\" (UID: \"65266674-4ef3-4abf-b20c-3c06447af305\") " pod="openshift-ingress-canary/ingress-canary-5x4l5"
Sep 09 19:55:24.939298 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-bound-sa-token\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.939298 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939199    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.939298 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kqs\" (UniqueName: \"kubernetes.io/projected/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-kube-api-access-z4kqs\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.939298 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-image-registry-private-configuration\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.939298 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgjpf\" (UniqueName: \"kubernetes.io/projected/65266674-4ef3-4abf-b20c-3c06447af305-kube-api-access-pgjpf\") pod \"ingress-canary-5x4l5\" (UID: \"65266674-4ef3-4abf-b20c-3c06447af305\") " pod="openshift-ingress-canary/ingress-canary-5x4l5"
Sep 09 19:55:24.939661 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-tmp-dir\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.939661 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.939661 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-ca-trust-extracted\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.939661 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-trusted-ca\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.939661 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrj\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-kube-api-access-p2zrj\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.939898 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.939768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-certificates\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.940470 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.939954    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:24.940470 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.940020    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls podName:eb6371e1-d79c-449d-a6a6-1db356e8fdf5 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:25.44000123 +0000 UTC m=+34.028654885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls") pod "dns-default-82kmf" (UID: "eb6371e1-d79c-449d-a6a6-1db356e8fdf5") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:24.940470 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.940106    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:24.940470 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.940118    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc687dd7-kf6hm: secret "image-registry-tls" not found
Sep 09 19:55:24.940470 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:24.940155    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls podName:76a619ca-d897-4baa-bf31-d0aed2ca4ffd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:25.440142061 +0000 UTC m=+34.028795699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls") pod "image-registry-55bc687dd7-kf6hm" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd") : secret "image-registry-tls" not found
Sep 09 19:55:24.940470 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.940195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-tmp-dir\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.940470 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.940261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-config-volume\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.940885 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.940799    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-ca-trust-extracted\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.941201 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.941173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-trusted-ca\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.944235 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.944213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-installation-pull-secrets\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.944341 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.944230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-image-registry-private-configuration\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.944431 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.944414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65266674-4ef3-4abf-b20c-3c06447af305-cert\") pod \"ingress-canary-5x4l5\" (UID: \"65266674-4ef3-4abf-b20c-3c06447af305\") " pod="openshift-ingress-canary/ingress-canary-5x4l5"
Sep 09 19:55:24.951947 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.951913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-bound-sa-token\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.952382 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.952360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kqs\" (UniqueName: \"kubernetes.io/projected/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-kube-api-access-z4kqs\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:24.952514 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.952499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2zrj\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-kube-api-access-p2zrj\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:24.953234 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.953213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgjpf\" (UniqueName: \"kubernetes.io/projected/65266674-4ef3-4abf-b20c-3c06447af305-kube-api-access-pgjpf\") pod \"ingress-canary-5x4l5\" (UID: \"65266674-4ef3-4abf-b20c-3c06447af305\") " pod="openshift-ingress-canary/ingress-canary-5x4l5"
Sep 09 19:55:24.995445 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:24.995409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:25.082124 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.082092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5x4l5"
Sep 09 19:55:25.444801 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.444767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:25.444993 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.444849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:25.444993 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:25.444963    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:25.444993 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:25.444976    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:25.444993 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:25.444985    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc687dd7-kf6hm: secret "image-registry-tls" not found
Sep 09 19:55:25.445169 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:25.445032    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls podName:eb6371e1-d79c-449d-a6a6-1db356e8fdf5 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:26.445013496 +0000 UTC m=+35.033667148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls") pod "dns-default-82kmf" (UID: "eb6371e1-d79c-449d-a6a6-1db356e8fdf5") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:25.445169 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:25.445057    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls podName:76a619ca-d897-4baa-bf31-d0aed2ca4ffd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:26.445038563 +0000 UTC m=+35.033692213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls") pod "image-registry-55bc687dd7-kf6hm" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd") : secret "image-registry-tls" not found
Sep 09 19:55:25.890648 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.890405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:25.890833 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.890404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:25.890833 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.890462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:25.893579 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.893538    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:55:25.893800 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.893781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:55:25.894477 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.894444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:55:25.894477 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.894472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:55:25.894722 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.894706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt99m\""
Sep 09 19:55:25.894810 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:25.894758    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-trbh8\""
Sep 09 19:55:26.452675 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:26.452626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:26.453139 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:26.452702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:26.453139 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:26.452787    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:26.453139 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:26.452859    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls podName:eb6371e1-d79c-449d-a6a6-1db356e8fdf5 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:28.452841039 +0000 UTC m=+37.041494689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls") pod "dns-default-82kmf" (UID: "eb6371e1-d79c-449d-a6a6-1db356e8fdf5") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:26.453139 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:26.452795    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:26.453139 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:26.452892    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc687dd7-kf6hm: secret "image-registry-tls" not found
Sep 09 19:55:26.453139 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:26.452972    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls podName:76a619ca-d897-4baa-bf31-d0aed2ca4ffd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:28.452954069 +0000 UTC m=+37.041607725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls") pod "image-registry-55bc687dd7-kf6hm" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd") : secret "image-registry-tls" not found
Sep 09 19:55:26.790519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:26.790489    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5x4l5"]
Sep 09 19:55:26.793741 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:26.793716    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65266674_4ef3_4abf_b20c_3c06447af305.slice/crio-33c1519271712a06a9a05f1094c7ad895ab1e7ecb8b791ea04745077dab612ac WatchSource:0}: Error finding container 33c1519271712a06a9a05f1094c7ad895ab1e7ecb8b791ea04745077dab612ac: Status 404 returned error can't find the container with id 33c1519271712a06a9a05f1094c7ad895ab1e7ecb8b791ea04745077dab612ac
Sep 09 19:55:27.072971 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:27.072897    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cd81c0-a0cd-48ee-a49b-5496fedaf6d8" containerID="a74a4bce30a4b01da5536ef726c5e7a6cb179427a451d5dec0f606b591bb1e85" exitCode=0
Sep 09 19:55:27.073123 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:27.072979    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b6dl" event={"ID":"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8","Type":"ContainerDied","Data":"a74a4bce30a4b01da5536ef726c5e7a6cb179427a451d5dec0f606b591bb1e85"}
Sep 09 19:55:27.074126 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:27.074105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5x4l5" event={"ID":"65266674-4ef3-4abf-b20c-3c06447af305","Type":"ContainerStarted","Data":"33c1519271712a06a9a05f1094c7ad895ab1e7ecb8b791ea04745077dab612ac"}
Sep 09 19:55:28.079292 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:28.079257    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cd81c0-a0cd-48ee-a49b-5496fedaf6d8" containerID="ce26ee17f040eed3f985fdeed412efb6816513108a4e024be009fe99df22e3c0" exitCode=0
Sep 09 19:55:28.080035 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:28.079301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b6dl" event={"ID":"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8","Type":"ContainerDied","Data":"ce26ee17f040eed3f985fdeed412efb6816513108a4e024be009fe99df22e3c0"}
Sep 09 19:55:28.472200 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:28.472010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:28.472352 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:28.472241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:28.472352 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:28.472176    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:28.472352 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:28.472282    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc687dd7-kf6hm: secret "image-registry-tls" not found
Sep 09 19:55:28.472352 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:28.472339    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:28.472516 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:28.472357    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls podName:76a619ca-d897-4baa-bf31-d0aed2ca4ffd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:32.472338465 +0000 UTC m=+41.060992102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls") pod "image-registry-55bc687dd7-kf6hm" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd") : secret "image-registry-tls" not found
Sep 09 19:55:28.472516 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:28.472377    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls podName:eb6371e1-d79c-449d-a6a6-1db356e8fdf5 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:32.472366417 +0000 UTC m=+41.061020076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls") pod "dns-default-82kmf" (UID: "eb6371e1-d79c-449d-a6a6-1db356e8fdf5") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:29.086158 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:29.086115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6b6dl" event={"ID":"37cd81c0-a0cd-48ee-a49b-5496fedaf6d8","Type":"ContainerStarted","Data":"c73364f77cf4599a66cbbaa37947ca8ba394fc5557491acea94913fa4b2c0068"}
Sep 09 19:55:29.126074 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:29.126018    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6b6dl" podStartSLOduration=10.883603835 podStartE2EDuration="37.125996115s" totalImagesPullingTime="26.24239228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:29.125002564 +0000 UTC m=+37.713656221" watchObservedRunningTime="2026-09-09 19:55:29.125996115 +0000 UTC m=+37.714649775"
Sep 09 19:55:29.273185 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:29.273151    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:29.340760 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:29.340729    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:30.090612 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:30.090581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5x4l5" event={"ID":"65266674-4ef3-4abf-b20c-3c06447af305","Type":"ContainerStarted","Data":"345d7191729b5c809435857ffd95c7a62b3898fe1c72eb3a7f6388561778f799"}
Sep 09 19:55:30.108949 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:30.108902    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5x4l5" podStartSLOduration=3.163492075 podStartE2EDuration="6.108887464s" totalImagesPullingTime="2.945395389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:30.107675743 +0000 UTC m=+38.696329399" watchObservedRunningTime="2026-09-09 19:55:30.108887464 +0000 UTC m=+38.697541115"
Sep 09 19:55:30.171194 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:30.171160    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:30.271289 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:30.271256    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:31.688018 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.687986    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"]
Sep 09 19:55:31.707618 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.707566    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"]
Sep 09 19:55:31.707781 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.707707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:31.712389 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.712365    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 19:55:31.712525 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.712455    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.712598 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.712527    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.712598 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.712566    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vslpk\""
Sep 09 19:55:31.730052 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.730027    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"]
Sep 09 19:55:31.730161 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.730061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"]
Sep 09 19:55:31.730161 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.730147    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:31.733823 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.733801    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 19:55:31.733917 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.733888    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6dvd4\""
Sep 09 19:55:31.734625 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.734608    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.734625 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.734611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.734897 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.734883    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 19:55:31.762393 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.762369    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6"]
Sep 09 19:55:31.787592 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.787567    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"]
Sep 09 19:55:31.787744 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.787708    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6"
Sep 09 19:55:31.791205 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.791162    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.791399 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.791328    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9s76b\""
Sep 09 19:55:31.791530 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.791516    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.797830 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.797811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:31.797925 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.797857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428j4\" (UniqueName: \"kubernetes.io/projected/c1910aa9-980d-49fe-be43-746579cb23e6-kube-api-access-428j4\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:31.811764 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.811739    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-s2v7q"]
Sep 09 19:55:31.811887 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.811871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:31.814686 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.814587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 19:55:31.814801 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.814724    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 19:55:31.814843 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.814800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-d7jrl\""
Sep 09 19:55:31.814843 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.814822    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.815028 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.814986    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.823983 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.823963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6"]
Sep 09 19:55:31.824080 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.824001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"]
Sep 09 19:55:31.824080 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.824020    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-s2v7q"]
Sep 09 19:55:31.824180 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.824104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:31.829235 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.829087    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cjd2l\""
Sep 09 19:55:31.829417 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.829363    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 19:55:31.829417 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.829382    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.829564 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.829450    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 19:55:31.829738 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.829725    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.836088 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.836071    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 19:55:31.898540 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08238358-ba3a-492a-b867-8edd7905ad48-serving-cert\") pod \"service-ca-operator-859f94c4b-6p9hx\" (UID: \"08238358-ba3a-492a-b867-8edd7905ad48\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:31.898727 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29f6\" (UniqueName: \"kubernetes.io/projected/82c99ae4-950c-4770-bd66-d8778fe3baf4-kube-api-access-r29f6\") pod \"kube-storage-version-migrator-operator-8d7dd4675-9ts9g\" (UID: \"82c99ae4-950c-4770-bd66-d8778fe3baf4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:31.898727 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fcff86d-6410-409d-975b-877aa305ceb1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:31.898727 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6h6m\" (UniqueName: \"kubernetes.io/projected/08e820ae-bca1-46fb-be7e-ef43f0fefdd5-kube-api-access-r6h6m\") pod \"volume-data-source-validator-695c895bd6-fm5k6\" (UID: \"08e820ae-bca1-46fb-be7e-ef43f0fefdd5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6"
Sep 09 19:55:31.898727 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898624    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fcff86d-6410-409d-975b-877aa305ceb1-tmp\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:31.898727 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428j4\" (UniqueName: \"kubernetes.io/projected/c1910aa9-980d-49fe-be43-746579cb23e6-kube-api-access-428j4\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:31.898727 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08238358-ba3a-492a-b867-8edd7905ad48-config\") pod \"service-ca-operator-859f94c4b-6p9hx\" (UID: \"08238358-ba3a-492a-b867-8edd7905ad48\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:31.899042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2clgd\" (UniqueName: \"kubernetes.io/projected/08238358-ba3a-492a-b867-8edd7905ad48-kube-api-access-2clgd\") pod \"service-ca-operator-859f94c4b-6p9hx\" (UID: \"08238358-ba3a-492a-b867-8edd7905ad48\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:31.899042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8fcff86d-6410-409d-975b-877aa305ceb1-snapshots\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:31.899042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:31.899042 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:31.898883    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:31.899042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntln6\" (UniqueName: \"kubernetes.io/projected/8fcff86d-6410-409d-975b-877aa305ceb1-kube-api-access-ntln6\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:31.899042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fcff86d-6410-409d-975b-877aa305ceb1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:31.899042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.898965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/82c99ae4-950c-4770-bd66-d8778fe3baf4-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-9ts9g\" (UID: \"82c99ae4-950c-4770-bd66-d8778fe3baf4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:31.899042 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:31.898995    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls podName:c1910aa9-980d-49fe-be43-746579cb23e6 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:32.398970883 +0000 UTC m=+40.987624517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jv4nv" (UID: "c1910aa9-980d-49fe-be43-746579cb23e6") : secret "samples-operator-tls" not found
Sep 09 19:55:31.899286 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.899074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fcff86d-6410-409d-975b-877aa305ceb1-serving-cert\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:31.899286 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.899105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/82c99ae4-950c-4770-bd66-d8778fe3baf4-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-9ts9g\" (UID: \"82c99ae4-950c-4770-bd66-d8778fe3baf4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:31.913454 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.913423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-428j4\" (UniqueName: \"kubernetes.io/projected/c1910aa9-980d-49fe-be43-746579cb23e6-kube-api-access-428j4\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:31.999893 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.999813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29f6\" (UniqueName: \"kubernetes.io/projected/82c99ae4-950c-4770-bd66-d8778fe3baf4-kube-api-access-r29f6\") pod \"kube-storage-version-migrator-operator-8d7dd4675-9ts9g\" (UID: \"82c99ae4-950c-4770-bd66-d8778fe3baf4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:31.999893 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.999849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fcff86d-6410-409d-975b-877aa305ceb1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:31.999893 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.999870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6h6m\" (UniqueName: \"kubernetes.io/projected/08e820ae-bca1-46fb-be7e-ef43f0fefdd5-kube-api-access-r6h6m\") pod \"volume-data-source-validator-695c895bd6-fm5k6\" (UID: \"08e820ae-bca1-46fb-be7e-ef43f0fefdd5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6"
Sep 09 19:55:31.999893 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.999888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fcff86d-6410-409d-975b-877aa305ceb1-tmp\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.000195 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.999945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08238358-ba3a-492a-b867-8edd7905ad48-config\") pod \"service-ca-operator-859f94c4b-6p9hx\" (UID: \"08238358-ba3a-492a-b867-8edd7905ad48\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:32.000195 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.999971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2clgd\" (UniqueName: \"kubernetes.io/projected/08238358-ba3a-492a-b867-8edd7905ad48-kube-api-access-2clgd\") pod \"service-ca-operator-859f94c4b-6p9hx\" (UID: \"08238358-ba3a-492a-b867-8edd7905ad48\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:32.000195 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:31.999995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8fcff86d-6410-409d-975b-877aa305ceb1-snapshots\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.000195 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntln6\" (UniqueName: \"kubernetes.io/projected/8fcff86d-6410-409d-975b-877aa305ceb1-kube-api-access-ntln6\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.000195 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fcff86d-6410-409d-975b-877aa305ceb1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.000463 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/82c99ae4-950c-4770-bd66-d8778fe3baf4-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-9ts9g\" (UID: \"82c99ae4-950c-4770-bd66-d8778fe3baf4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:32.000463 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fcff86d-6410-409d-975b-877aa305ceb1-serving-cert\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.000463 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/82c99ae4-950c-4770-bd66-d8778fe3baf4-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-9ts9g\" (UID: \"82c99ae4-950c-4770-bd66-d8778fe3baf4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:32.000463 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000294    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08238358-ba3a-492a-b867-8edd7905ad48-serving-cert\") pod \"service-ca-operator-859f94c4b-6p9hx\" (UID: \"08238358-ba3a-492a-b867-8edd7905ad48\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:32.000691 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8fcff86d-6410-409d-975b-877aa305ceb1-snapshots\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.000755 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08238358-ba3a-492a-b867-8edd7905ad48-config\") pod \"service-ca-operator-859f94c4b-6p9hx\" (UID: \"08238358-ba3a-492a-b867-8edd7905ad48\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:32.000913 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fcff86d-6410-409d-975b-877aa305ceb1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.001039 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.000992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fcff86d-6410-409d-975b-877aa305ceb1-tmp\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.001039 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.001004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/82c99ae4-950c-4770-bd66-d8778fe3baf4-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-9ts9g\" (UID: \"82c99ae4-950c-4770-bd66-d8778fe3baf4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:32.002626 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.002600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/82c99ae4-950c-4770-bd66-d8778fe3baf4-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-9ts9g\" (UID: \"82c99ae4-950c-4770-bd66-d8778fe3baf4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:32.002708 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.002680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08238358-ba3a-492a-b867-8edd7905ad48-serving-cert\") pod \"service-ca-operator-859f94c4b-6p9hx\" (UID: \"08238358-ba3a-492a-b867-8edd7905ad48\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:32.002708 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.002701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8fcff86d-6410-409d-975b-877aa305ceb1-serving-cert\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.008567 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.008540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntln6\" (UniqueName: \"kubernetes.io/projected/8fcff86d-6410-409d-975b-877aa305ceb1-kube-api-access-ntln6\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.008706 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.008614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29f6\" (UniqueName: \"kubernetes.io/projected/82c99ae4-950c-4770-bd66-d8778fe3baf4-kube-api-access-r29f6\") pod \"kube-storage-version-migrator-operator-8d7dd4675-9ts9g\" (UID: \"82c99ae4-950c-4770-bd66-d8778fe3baf4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:32.008893 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.008868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2clgd\" (UniqueName: \"kubernetes.io/projected/08238358-ba3a-492a-b867-8edd7905ad48-kube-api-access-2clgd\") pod \"service-ca-operator-859f94c4b-6p9hx\" (UID: \"08238358-ba3a-492a-b867-8edd7905ad48\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:32.009471 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.009450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6h6m\" (UniqueName: \"kubernetes.io/projected/08e820ae-bca1-46fb-be7e-ef43f0fefdd5-kube-api-access-r6h6m\") pod \"volume-data-source-validator-695c895bd6-fm5k6\" (UID: \"08e820ae-bca1-46fb-be7e-ef43f0fefdd5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6"
Sep 09 19:55:32.010383 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.010360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fcff86d-6410-409d-975b-877aa305ceb1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-s2v7q\" (UID: \"8fcff86d-6410-409d-975b-877aa305ceb1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.038548 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.038519    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"
Sep 09 19:55:32.044783 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.044753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:32.096414 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.096379    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6"
Sep 09 19:55:32.120595 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.120570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"
Sep 09 19:55:32.131999 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.131981    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q"
Sep 09 19:55:32.146070 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.146042    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:32.245909 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.245871    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:32.320842 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.320818    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g"]
Sep 09 19:55:32.323203 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:32.323177    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c99ae4_950c_4770_bd66_d8778fe3baf4.slice/crio-2aa75ae403745377ebdc267558cdbad452ac6335e405f4f30b31c2feaea6cb55 WatchSource:0}: Error finding container 2aa75ae403745377ebdc267558cdbad452ac6335e405f4f30b31c2feaea6cb55: Status 404 returned error can't find the container with id 2aa75ae403745377ebdc267558cdbad452ac6335e405f4f30b31c2feaea6cb55
Sep 09 19:55:32.384982 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.384953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6"]
Sep 09 19:55:32.398936 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:32.398906    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e820ae_bca1_46fb_be7e_ef43f0fefdd5.slice/crio-4039821fe80d0055187f60f534bf3089e630eaa6195926001c3e3eb53de99420 WatchSource:0}: Error finding container 4039821fe80d0055187f60f534bf3089e630eaa6195926001c3e3eb53de99420: Status 404 returned error can't find the container with id 4039821fe80d0055187f60f534bf3089e630eaa6195926001c3e3eb53de99420
Sep 09 19:55:32.403585 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.403563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:32.403727 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:32.403709    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:32.403776 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:32.403765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls podName:c1910aa9-980d-49fe-be43-746579cb23e6 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.403750362 +0000 UTC m=+41.992403996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jv4nv" (UID: "c1910aa9-980d-49fe-be43-746579cb23e6") : secret "samples-operator-tls" not found
Sep 09 19:55:32.412843 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.412819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx"]
Sep 09 19:55:32.415992 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:32.415970    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08238358_ba3a_492a_b867_8edd7905ad48.slice/crio-6daf9d8b5f696acd112e4d2587b1c6dad32ebacfc4e01221d6162215837c97de WatchSource:0}: Error finding container 6daf9d8b5f696acd112e4d2587b1c6dad32ebacfc4e01221d6162215837c97de: Status 404 returned error can't find the container with id 6daf9d8b5f696acd112e4d2587b1c6dad32ebacfc4e01221d6162215837c97de
Sep 09 19:55:32.455402 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.455369    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-s2v7q"]
Sep 09 19:55:32.458081 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:32.458060    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fcff86d_6410_409d_975b_877aa305ceb1.slice/crio-9049f97aac47b9b8dd40e8183e10b20f348fff35ac3841dbd0166986e0204536 WatchSource:0}: Error finding container 9049f97aac47b9b8dd40e8183e10b20f348fff35ac3841dbd0166986e0204536: Status 404 returned error can't find the container with id 9049f97aac47b9b8dd40e8183e10b20f348fff35ac3841dbd0166986e0204536
Sep 09 19:55:32.504060 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.504018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:32.504169 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:32.504096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:32.504210 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:32.504162    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:32.504210 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:32.504180    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc687dd7-kf6hm: secret "image-registry-tls" not found
Sep 09 19:55:32.504210 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:32.504184    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:32.504299 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:32.504229    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls podName:76a619ca-d897-4baa-bf31-d0aed2ca4ffd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:40.504213819 +0000 UTC m=+49.092867470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls") pod "image-registry-55bc687dd7-kf6hm" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd") : secret "image-registry-tls" not found
Sep 09 19:55:32.504299 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:32.504243    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls podName:eb6371e1-d79c-449d-a6a6-1db356e8fdf5 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:40.504236952 +0000 UTC m=+49.092890607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls") pod "dns-default-82kmf" (UID: "eb6371e1-d79c-449d-a6a6-1db356e8fdf5") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:33.098512 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:33.098468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q" event={"ID":"8fcff86d-6410-409d-975b-877aa305ceb1","Type":"ContainerStarted","Data":"9049f97aac47b9b8dd40e8183e10b20f348fff35ac3841dbd0166986e0204536"}
Sep 09 19:55:33.099778 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:33.099751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx" event={"ID":"08238358-ba3a-492a-b867-8edd7905ad48","Type":"ContainerStarted","Data":"6daf9d8b5f696acd112e4d2587b1c6dad32ebacfc4e01221d6162215837c97de"}
Sep 09 19:55:33.101050 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:33.101027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6" event={"ID":"08e820ae-bca1-46fb-be7e-ef43f0fefdd5","Type":"ContainerStarted","Data":"4039821fe80d0055187f60f534bf3089e630eaa6195926001c3e3eb53de99420"}
Sep 09 19:55:33.102214 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:33.102192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g" event={"ID":"82c99ae4-950c-4770-bd66-d8778fe3baf4","Type":"ContainerStarted","Data":"2aa75ae403745377ebdc267558cdbad452ac6335e405f4f30b31c2feaea6cb55"}
Sep 09 19:55:33.410784 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:33.410699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:33.410940 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:33.410874    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:33.411006 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:33.410954    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls podName:c1910aa9-980d-49fe-be43-746579cb23e6 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:35.4109326 +0000 UTC m=+43.999586255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jv4nv" (UID: "c1910aa9-980d-49fe-be43-746579cb23e6") : secret "samples-operator-tls" not found
Sep 09 19:55:34.332636 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:34.332595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:34.403795 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:34.403760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:35.333297 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:35.333266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:35.428050 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:35.427915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:35.428213 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:35.428057    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:35.428213 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:35.428140    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls podName:c1910aa9-980d-49fe-be43-746579cb23e6 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.428119797 +0000 UTC m=+48.016773466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jv4nv" (UID: "c1910aa9-980d-49fe-be43-746579cb23e6") : secret "samples-operator-tls" not found
Sep 09 19:55:35.449335 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:35.449288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:36.288686 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:36.288655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:36.357034 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:36.357003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:36.457484 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:36.457462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:36.557424 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:36.557354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:37.116425 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.116385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q" event={"ID":"8fcff86d-6410-409d-975b-877aa305ceb1","Type":"ContainerStarted","Data":"3474d15f72c8246e17d65286af649dc6c6756a5332ccd14ffdd6e1ed81f20e01"}
Sep 09 19:55:37.118292 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.118261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx" event={"ID":"08238358-ba3a-492a-b867-8edd7905ad48","Type":"ContainerStarted","Data":"1ff96cad60d95dc5f4ba6c1a30642f6ed02969d314664ce125049f7f6500ff6f"}
Sep 09 19:55:37.119966 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.119902    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6" event={"ID":"08e820ae-bca1-46fb-be7e-ef43f0fefdd5","Type":"ContainerStarted","Data":"595f6247a821519236233c50c7bc06aefe3948cdeece857be04c2c188c48e55a"}
Sep 09 19:55:37.121620 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.121598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g" event={"ID":"82c99ae4-950c-4770-bd66-d8778fe3baf4","Type":"ContainerStarted","Data":"862e2bb1db6a54ca0400d8b27002bd23e55f95810b51b035b1d3701efc59c1ef"}
Sep 09 19:55:37.134074 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.134021    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q" podStartSLOduration=1.9696003979999999 podStartE2EDuration="6.134004388s" totalImagesPullingTime="4.16440399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:37.132860829 +0000 UTC m=+45.721514487" watchObservedRunningTime="2026-09-09 19:55:37.134004388 +0000 UTC m=+45.722658039"
Sep 09 19:55:37.147265 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.147017    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx" podStartSLOduration=1.940789481 podStartE2EDuration="6.147004093s" totalImagesPullingTime="4.206214612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:37.146648443 +0000 UTC m=+45.735302100" watchObservedRunningTime="2026-09-09 19:55:37.147004093 +0000 UTC m=+45.735657812"
Sep 09 19:55:37.167294 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.167249    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g" podStartSLOduration=1.860346763 podStartE2EDuration="6.167235381s" totalImagesPullingTime="4.306888618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:37.166821636 +0000 UTC m=+45.755475312" watchObservedRunningTime="2026-09-09 19:55:37.167235381 +0000 UTC m=+45.755889051"
Sep 09 19:55:37.188264 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.188212    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-fm5k6" podStartSLOduration=1.988474188 podStartE2EDuration="6.188197882s" totalImagesPullingTime="4.199723694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:37.187596705 +0000 UTC m=+45.776250362" watchObservedRunningTime="2026-09-09 19:55:37.188197882 +0000 UTC m=+45.776851539"
Sep 09 19:55:37.322728 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.322691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:37.426696 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.426595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:37.856715 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.856682    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:37.955966 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:37.955936    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:38.015607 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.015571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:38.511229 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.511197    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mzd2f"]
Sep 09 19:55:38.530845 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.530808    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:38.531847 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.531826    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mzd2f"]
Sep 09 19:55:38.531957 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.531940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.534704 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.534673    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 19:55:38.534819 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.534673    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 19:55:38.534819 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.534722    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zhnk7\""
Sep 09 19:55:38.600915 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.600884    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:38.656120 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.656086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.656265 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.656137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aadad933-4edc-411c-9787-38eea928a954-crio-socket\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.656265 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.656198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxvw\" (UniqueName: \"kubernetes.io/projected/aadad933-4edc-411c-9787-38eea928a954-kube-api-access-fxxvw\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.656391 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.656303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aadad933-4edc-411c-9787-38eea928a954-data-volume\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.656391 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.656344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.675077 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.675048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:38.757341 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.757295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aadad933-4edc-411c-9787-38eea928a954-data-volume\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.757341 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.757343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.757531 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.757374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.757531 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.757401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aadad933-4edc-411c-9787-38eea928a954-crio-socket\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.757531 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.757419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxvw\" (UniqueName: \"kubernetes.io/projected/aadad933-4edc-411c-9787-38eea928a954-kube-api-access-fxxvw\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.757531 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:38.757486    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:38.757531 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:38.757507    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:38.757734 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:38.757559    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm podName:aadad933-4edc-411c-9787-38eea928a954 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.257539239 +0000 UTC m=+47.846192888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mzd2f" (UID: "aadad933-4edc-411c-9787-38eea928a954") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:38.757734 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:38.757580    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls podName:aadad933-4edc-411c-9787-38eea928a954 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.257571082 +0000 UTC m=+47.846224755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzd2f" (UID: "aadad933-4edc-411c-9787-38eea928a954") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:38.757734 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.757638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aadad933-4edc-411c-9787-38eea928a954-crio-socket\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.757734 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.757655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aadad933-4edc-411c-9787-38eea928a954-data-volume\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.765730 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.765679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxvw\" (UniqueName: \"kubernetes.io/projected/aadad933-4edc-411c-9787-38eea928a954-kube-api-access-fxxvw\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:38.848404 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.848375    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg"]
Sep 09 19:55:38.870920 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.870897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg"]
Sep 09 19:55:38.871036 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.870993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg"
Sep 09 19:55:38.874365 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.874346    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-khdrm\""
Sep 09 19:55:38.874515 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.874500    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 19:55:38.874720 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.874706    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:38.894738 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.894707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:38.959703 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.959670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99k6p\" (UniqueName: \"kubernetes.io/projected/c55bb2b8-6bb7-436e-86e2-f517432078cb-kube-api-access-99k6p\") pod \"migrator-fdbb8fdff-fspqg\" (UID: \"c55bb2b8-6bb7-436e-86e2-f517432078cb\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg"
Sep 09 19:55:38.962006 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:38.961971    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:39.060505 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:39.060434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99k6p\" (UniqueName: \"kubernetes.io/projected/c55bb2b8-6bb7-436e-86e2-f517432078cb-kube-api-access-99k6p\") pod \"migrator-fdbb8fdff-fspqg\" (UID: \"c55bb2b8-6bb7-436e-86e2-f517432078cb\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg"
Sep 09 19:55:39.071688 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:39.071661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99k6p\" (UniqueName: \"kubernetes.io/projected/c55bb2b8-6bb7-436e-86e2-f517432078cb-kube-api-access-99k6p\") pod \"migrator-fdbb8fdff-fspqg\" (UID: \"c55bb2b8-6bb7-436e-86e2-f517432078cb\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg"
Sep 09 19:55:39.180039 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:39.180006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg"
Sep 09 19:55:39.262028 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:39.261992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:39.262131 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:39.262046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:39.262173 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:39.262151    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:39.262207 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:39.262200    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm podName:aadad933-4edc-411c-9787-38eea928a954 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:40.262184813 +0000 UTC m=+48.850838448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mzd2f" (UID: "aadad933-4edc-411c-9787-38eea928a954") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:39.262381 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:39.262365    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:39.262493 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:39.262478    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls podName:aadad933-4edc-411c-9787-38eea928a954 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:40.262460519 +0000 UTC m=+48.851114173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzd2f" (UID: "aadad933-4edc-411c-9787-38eea928a954") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:39.463917 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:39.463883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:39.464069 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:39.464032    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:39.464106 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:39.464092    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls podName:c1910aa9-980d-49fe-be43-746579cb23e6 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.464076982 +0000 UTC m=+56.052730617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jv4nv" (UID: "c1910aa9-980d-49fe-be43-746579cb23e6") : secret "samples-operator-tls" not found
Sep 09 19:55:39.466008 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:39.465984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg"]
Sep 09 19:55:39.468586 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:39.468568    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55bb2b8_6bb7_436e_86e2_f517432078cb.slice/crio-7599c486c6ca86b3b6a2f144d02640e4b058bdd91334306519951759e0102077 WatchSource:0}: Error finding container 7599c486c6ca86b3b6a2f144d02640e4b058bdd91334306519951759e0102077: Status 404 returned error can't find the container with id 7599c486c6ca86b3b6a2f144d02640e4b058bdd91334306519951759e0102077
Sep 09 19:55:40.131905 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.131865    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg" event={"ID":"c55bb2b8-6bb7-436e-86e2-f517432078cb","Type":"ContainerStarted","Data":"7599c486c6ca86b3b6a2f144d02640e4b058bdd91334306519951759e0102077"}
Sep 09 19:55:40.246138 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.246107    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-jddjx"]
Sep 09 19:55:40.261768 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.261720    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-jddjx"]
Sep 09 19:55:40.261936 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.261852    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.264717 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.264532    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 19:55:40.265058 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.265014    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pm4jm\""
Sep 09 19:55:40.265734 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.265706    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 19:55:40.265873 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.265850    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 19:55:40.265970 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.265935    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 19:55:40.272272 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.272247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:40.272392 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.272325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:40.272455 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:40.272422    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:40.272455 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:40.272431    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:40.272557 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:40.272482    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm podName:aadad933-4edc-411c-9787-38eea928a954 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:42.272464618 +0000 UTC m=+50.861118261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mzd2f" (UID: "aadad933-4edc-411c-9787-38eea928a954") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:40.272557 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:40.272498    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls podName:aadad933-4edc-411c-9787-38eea928a954 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:42.272489344 +0000 UTC m=+50.861142981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzd2f" (UID: "aadad933-4edc-411c-9787-38eea928a954") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:40.293640 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.293619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ztrxp_cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6/dns-node-resolver/0.log"
Sep 09 19:55:40.335604 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.335574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:40.373231 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.373202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjcc\" (UniqueName: \"kubernetes.io/projected/2a6f8096-2e8d-4372-86f1-aa2dfa6132f3-kube-api-access-dsjcc\") pod \"service-ca-75d99c484-jddjx\" (UID: \"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3\") " pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.373421 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.373260    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2a6f8096-2e8d-4372-86f1-aa2dfa6132f3-signing-key\") pod \"service-ca-75d99c484-jddjx\" (UID: \"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3\") " pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.373421 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.373416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a6f8096-2e8d-4372-86f1-aa2dfa6132f3-signing-cabundle\") pod \"service-ca-75d99c484-jddjx\" (UID: \"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3\") " pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.397622 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.397536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:40.474793 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.474763    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2a6f8096-2e8d-4372-86f1-aa2dfa6132f3-signing-key\") pod \"service-ca-75d99c484-jddjx\" (UID: \"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3\") " pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.474939 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.474834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a6f8096-2e8d-4372-86f1-aa2dfa6132f3-signing-cabundle\") pod \"service-ca-75d99c484-jddjx\" (UID: \"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3\") " pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.474939 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.474896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjcc\" (UniqueName: \"kubernetes.io/projected/2a6f8096-2e8d-4372-86f1-aa2dfa6132f3-kube-api-access-dsjcc\") pod \"service-ca-75d99c484-jddjx\" (UID: \"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3\") " pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.475535 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.475511    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a6f8096-2e8d-4372-86f1-aa2dfa6132f3-signing-cabundle\") pod \"service-ca-75d99c484-jddjx\" (UID: \"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3\") " pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.477133 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.477109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2a6f8096-2e8d-4372-86f1-aa2dfa6132f3-signing-key\") pod \"service-ca-75d99c484-jddjx\" (UID: \"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3\") " pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.483026 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.482998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjcc\" (UniqueName: \"kubernetes.io/projected/2a6f8096-2e8d-4372-86f1-aa2dfa6132f3-kube-api-access-dsjcc\") pod \"service-ca-75d99c484-jddjx\" (UID: \"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3\") " pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.573645 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.573614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-jddjx"
Sep 09 19:55:40.575997 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.575974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:40.576107 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.576035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:40.576152 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:40.576118    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:40.576194 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:40.576170    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls podName:eb6371e1-d79c-449d-a6a6-1db356e8fdf5 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.576156218 +0000 UTC m=+65.164809853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls") pod "dns-default-82kmf" (UID: "eb6371e1-d79c-449d-a6a6-1db356e8fdf5") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:40.576241 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:40.576216    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:40.576241 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:40.576235    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bc687dd7-kf6hm: secret "image-registry-tls" not found
Sep 09 19:55:40.576341 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:40.576279    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls podName:76a619ca-d897-4baa-bf31-d0aed2ca4ffd nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.576267538 +0000 UTC m=+65.164921172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls") pod "image-registry-55bc687dd7-kf6hm" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd") : secret "image-registry-tls" not found
Sep 09 19:55:40.708340 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:40.708225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:41.084843 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:41.084776    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-jddjx"]
Sep 09 19:55:41.157223 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:41.157196    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6f8096_2e8d_4372_86f1_aa2dfa6132f3.slice/crio-0c571d5b5a9e35816e7a6daaf42aa1024c670740b954c4027ecf0be355eb0331 WatchSource:0}: Error finding container 0c571d5b5a9e35816e7a6daaf42aa1024c670740b954c4027ecf0be355eb0331: Status 404 returned error can't find the container with id 0c571d5b5a9e35816e7a6daaf42aa1024c670740b954c4027ecf0be355eb0331
Sep 09 19:55:41.307218 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:41.307195    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rxfgd_175a2b7e-f1c9-4a5c-bc4c-75f73cc79808/node-ca/0.log"
Sep 09 19:55:41.361365 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:41.361338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:41.696375 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:41.696286    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5x4l5_65266674-4ef3-4abf-b20c-3c06447af305/serve-healthcheck-canary/0.log"
Sep 09 19:55:42.142379 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:42.142339    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg" event={"ID":"c55bb2b8-6bb7-436e-86e2-f517432078cb","Type":"ContainerStarted","Data":"831353d5d6649af853c3d4d89d37a1218e456f9bbb161d24020533770a826418"}
Sep 09 19:55:42.142582 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:42.142403    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg" event={"ID":"c55bb2b8-6bb7-436e-86e2-f517432078cb","Type":"ContainerStarted","Data":"e905b68fdef66608f3791f2b3fcd95095c379e74c67fdcd79fe6982df4fce22e"}
Sep 09 19:55:42.144039 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:42.144013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-jddjx" event={"ID":"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3","Type":"ContainerStarted","Data":"8e560644ba6d133af8f9872e6d9ba6d82ca99f1b63acaa865ec88efe496a85c3"}
Sep 09 19:55:42.144149 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:42.144052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-jddjx" event={"ID":"2a6f8096-2e8d-4372-86f1-aa2dfa6132f3","Type":"ContainerStarted","Data":"0c571d5b5a9e35816e7a6daaf42aa1024c670740b954c4027ecf0be355eb0331"}
Sep 09 19:55:42.161057 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:42.161016    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fspqg" podStartSLOduration=2.4478790679999998 podStartE2EDuration="4.16100365s" totalImagesPullingTime="1.713124582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:42.160908312 +0000 UTC m=+50.749561968" watchObservedRunningTime="2026-09-09 19:55:42.16100365 +0000 UTC m=+50.749657306"
Sep 09 19:55:42.180333 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:42.180267    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-jddjx" podStartSLOduration=2.180249799 podStartE2EDuration="2.180249799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:42.17942055 +0000 UTC m=+50.768074200" watchObservedRunningTime="2026-09-09 19:55:42.180249799 +0000 UTC m=+50.768903457"
Sep 09 19:55:42.291273 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:42.291189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:42.291273 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:42.291245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:42.291471 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:42.291374    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:42.291471 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:42.291460    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls podName:aadad933-4edc-411c-9787-38eea928a954 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:46.291441866 +0000 UTC m=+54.880095519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzd2f" (UID: "aadad933-4edc-411c-9787-38eea928a954") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:42.291742 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:42.291725    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aadad933-4edc-411c-9787-38eea928a954-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:43.017355 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:43.017302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:43.228497 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:43.228466    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:43.329923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:43.329831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:43.439243 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:43.439212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:43.539262 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:43.539228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:43.746068 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:43.746040    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:44.470371 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:44.470337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:44.721324 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:44.721227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:45.127114 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:45.127081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:45.326112 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:45.326077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:45.532481 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:45.532396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:45.917687 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:45.917658    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:46.325768 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:46.325676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:46.325989 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:46.325845    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:46.325989 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:55:46.325908    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls podName:aadad933-4edc-411c-9787-38eea928a954 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:54.325891883 +0000 UTC m=+62.914545537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mzd2f" (UID: "aadad933-4edc-411c-9787-38eea928a954") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:46.517898 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:46.517861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:46.793277 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:46.793242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:46.850716 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:46.850688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:47.130845 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:47.130815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:47.191430 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:47.191395    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:47.533421 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:47.533337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:47.535817 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:47.535793    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c1910aa9-980d-49fe-be43-746579cb23e6-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jv4nv\" (UID: \"c1910aa9-980d-49fe-be43-746579cb23e6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:47.617990 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:47.617956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"
Sep 09 19:55:47.888789 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:47.888754    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv"]
Sep 09 19:55:47.919336 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:47.919293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:48.000614 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:48.000578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:48.123448 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:48.123414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:48.170280 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:48.170197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv" event={"ID":"c1910aa9-980d-49fe-be43-746579cb23e6","Type":"ContainerStarted","Data":"3f70ce2df11e2d0f69f1bcef63ede58445fd406eff98d4357fef00ad620cb309"}
Sep 09 19:55:48.336927 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:48.336893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:50.735717 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:50.735683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:50.796659 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:50.796634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:51.064976 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:51.064904    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ccdvx"
Sep 09 19:55:51.526778 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:51.526741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:51.955022 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:51.954978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:52.262688 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:52.262647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:52.361970 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:52.361940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:52.518068 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:52.518034    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:53.189249 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:53.189207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv" event={"ID":"c1910aa9-980d-49fe-be43-746579cb23e6","Type":"ContainerStarted","Data":"8582f6458b9c9c55fbea9417c4ebab0250572863525efe65397524ee45d7a4fe"}
Sep 09 19:55:53.189629 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:53.189271    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv" event={"ID":"c1910aa9-980d-49fe-be43-746579cb23e6","Type":"ContainerStarted","Data":"960c32ffb952e8724930726e6327a7a04a630a0e820c741071597dce9de4f1ab"}
Sep 09 19:55:53.206244 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:53.206200    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jv4nv" podStartSLOduration=17.748436374 podStartE2EDuration="22.206186269s" totalImagesPullingTime="4.457749895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:53.205706136 +0000 UTC m=+61.794359794" watchObservedRunningTime="2026-09-09 19:55:53.206186269 +0000 UTC m=+61.794839925"
Sep 09 19:55:53.321945 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:53.321918    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:53.517503 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:53.517426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:53.654732 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:53.654697    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:53.755849 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:53.755815    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:53.861983 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:53.861950    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:54.392081 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:54.392033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:54.394430 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:54.394392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aadad933-4edc-411c-9787-38eea928a954-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mzd2f\" (UID: \"aadad933-4edc-411c-9787-38eea928a954\") " pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:54.444851 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:54.444819    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zhnk7\""
Sep 09 19:55:54.452210 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:54.452188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mzd2f"
Sep 09 19:55:54.732109 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:54.732056    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mzd2f"]
Sep 09 19:55:54.734569 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:54.734534    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadad933_4edc_411c_9787_38eea928a954.slice/crio-221a44be2d9272aaa3e183b8f25b8d10d125e5fd87bef127fa3d54a70f8438b1 WatchSource:0}: Error finding container 221a44be2d9272aaa3e183b8f25b8d10d125e5fd87bef127fa3d54a70f8438b1: Status 404 returned error can't find the container with id 221a44be2d9272aaa3e183b8f25b8d10d125e5fd87bef127fa3d54a70f8438b1
Sep 09 19:55:55.174965 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:55.174940    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:55.197741 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:55.197711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mzd2f" event={"ID":"aadad933-4edc-411c-9787-38eea928a954","Type":"ContainerStarted","Data":"c2cabb664e77ddba2823e84ad5a9d342a15175286cff3b00d4efb7ec9e615b9d"}
Sep 09 19:55:55.197869 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:55.197764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mzd2f" event={"ID":"aadad933-4edc-411c-9787-38eea928a954","Type":"ContainerStarted","Data":"221a44be2d9272aaa3e183b8f25b8d10d125e5fd87bef127fa3d54a70f8438b1"}
Sep 09 19:55:56.203728 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.203684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mzd2f" event={"ID":"aadad933-4edc-411c-9787-38eea928a954","Type":"ContainerStarted","Data":"1147250b6751023dd8f31c94e6eb11b2c06fef489a0779daee004cf8a8b592df"}
Sep 09 19:55:56.609479 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.609445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:56.609665 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.609506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:56.609665 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.609555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:56.612014 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.611969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb6371e1-d79c-449d-a6a6-1db356e8fdf5-metrics-tls\") pod \"dns-default-82kmf\" (UID: \"eb6371e1-d79c-449d-a6a6-1db356e8fdf5\") " pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:56.612150 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.612112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"image-registry-55bc687dd7-kf6hm\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") " pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:56.612213 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.612195    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:55:56.622559 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.622537    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:55:56.632936 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.632912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vd5\" (UniqueName: \"kubernetes.io/projected/bc10e010-ba02-48e7-acec-96938127af83-kube-api-access-x8vd5\") pod \"network-check-target-t7mhh\" (UID: \"bc10e010-ba02-48e7-acec-96938127af83\") " pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:56.710243 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.710210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:56.710243 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.710249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:56.715237 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.715204    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:55:56.715237 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.715231    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:55:56.723026 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.722996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/34db695b-0f6e-43c2-b92f-65214c6f28e4-original-pull-secret\") pod \"global-pull-secret-syncer-2458j\" (UID: \"34db695b-0f6e-43c2-b92f-65214c6f28e4\") " pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:56.723026 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.723022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bb2809e-c82b-4a2b-86e6-dc341e864dfd-metrics-certs\") pod \"network-metrics-daemon-8r42k\" (UID: \"7bb2809e-c82b-4a2b-86e6-dc341e864dfd\") " pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:56.775110 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.775077    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:56.802390 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.802357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2458j"
Sep 09 19:55:56.811155 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.811133    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-trbh8\""
Sep 09 19:55:56.816016 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.815991    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt99m\""
Sep 09 19:55:56.819769 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.819750    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:55:56.824630 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.824611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8r42k"
Sep 09 19:55:56.849907 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.849878    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xdvvq\""
Sep 09 19:55:56.857866 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.857843    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:56.863378 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.863361    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5xtg\""
Sep 09 19:55:56.871231 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.871194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82kmf"
Sep 09 19:55:56.876088 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.876067    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:56.976620 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:56.976585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:57.209426 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.209390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mzd2f" event={"ID":"aadad933-4edc-411c-9787-38eea928a954","Type":"ContainerStarted","Data":"22e0a33a88546c9c10a5e767be0ce0561647b450ae70f25489e6f1cad7fd94f7"}
Sep 09 19:55:57.244927 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.244875    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mzd2f" podStartSLOduration=16.996596369 podStartE2EDuration="19.244859028s" totalImagesPullingTime="2.248262659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:57.244859283 +0000 UTC m=+65.833512951" watchObservedRunningTime="2026-09-09 19:55:57.244859028 +0000 UTC m=+65.833512685"
Sep 09 19:55:57.296171 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.296112    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:57.356408 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.356381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7mhh"]
Sep 09 19:55:57.359945 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:57.359899    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc10e010_ba02_48e7_acec_96938127af83.slice/crio-8d216f2bcf736d707ef77d8499b4fab40dfd19ea6633ec35e8e1b7f88bd212ab WatchSource:0}: Error finding container 8d216f2bcf736d707ef77d8499b4fab40dfd19ea6633ec35e8e1b7f88bd212ab: Status 404 returned error can't find the container with id 8d216f2bcf736d707ef77d8499b4fab40dfd19ea6633ec35e8e1b7f88bd212ab
Sep 09 19:55:57.361331 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.361175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:57.361331 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.361256    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8r42k"]
Sep 09 19:55:57.362372 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.362353    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-82kmf"]
Sep 09 19:55:57.363458 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.363380    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2458j"]
Sep 09 19:55:57.364191 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:57.364168    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb2809e_c82b_4a2b_86e6_dc341e864dfd.slice/crio-a5935e6834237944bed6255a57911298b4a65c62f3706e3c32dbc2658b658505 WatchSource:0}: Error finding container a5935e6834237944bed6255a57911298b4a65c62f3706e3c32dbc2658b658505: Status 404 returned error can't find the container with id a5935e6834237944bed6255a57911298b4a65c62f3706e3c32dbc2658b658505
Sep 09 19:55:57.366352 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.366328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bc687dd7-kf6hm"]
Sep 09 19:55:57.370869 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:55:57.370807    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a619ca_d897_4baa_bf31_d0aed2ca4ffd.slice/crio-d9ce9227b9342dc303444a158196d38ab8701ca85fc39c0d10b9c1d28c2158dd WatchSource:0}: Error finding container d9ce9227b9342dc303444a158196d38ab8701ca85fc39c0d10b9c1d28c2158dd: Status 404 returned error can't find the container with id d9ce9227b9342dc303444a158196d38ab8701ca85fc39c0d10b9c1d28c2158dd
Sep 09 19:55:57.457604 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.457576    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:57.553507 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:57.553477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:58.214733 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.214699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2458j" event={"ID":"34db695b-0f6e-43c2-b92f-65214c6f28e4","Type":"ContainerStarted","Data":"e27d4e8ea74d8ee35004daa1de930d63c65c9f6b9246e346d7c990883e993851"}
Sep 09 19:55:58.217141 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.216653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm" event={"ID":"76a619ca-d897-4baa-bf31-d0aed2ca4ffd","Type":"ContainerStarted","Data":"784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93"}
Sep 09 19:55:58.217141 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.216700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm" event={"ID":"76a619ca-d897-4baa-bf31-d0aed2ca4ffd","Type":"ContainerStarted","Data":"d9ce9227b9342dc303444a158196d38ab8701ca85fc39c0d10b9c1d28c2158dd"}
Sep 09 19:55:58.217141 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.216769    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:55:58.218227 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.218206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7mhh" event={"ID":"bc10e010-ba02-48e7-acec-96938127af83","Type":"ContainerStarted","Data":"8d216f2bcf736d707ef77d8499b4fab40dfd19ea6633ec35e8e1b7f88bd212ab"}
Sep 09 19:55:58.219524 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.219501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82kmf" event={"ID":"eb6371e1-d79c-449d-a6a6-1db356e8fdf5","Type":"ContainerStarted","Data":"acb75ffaac8f7ec39c9743513b68f0e170130c44f1af1a079ba9da6b10a8d99a"}
Sep 09 19:55:58.220928 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.220903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8r42k" event={"ID":"7bb2809e-c82b-4a2b-86e6-dc341e864dfd","Type":"ContainerStarted","Data":"a5935e6834237944bed6255a57911298b4a65c62f3706e3c32dbc2658b658505"}
Sep 09 19:55:58.247192 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.247135    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm" podStartSLOduration=66.247116959 podStartE2EDuration="1m6.247116959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:58.246335277 +0000 UTC m=+66.834988934" watchObservedRunningTime="2026-09-09 19:55:58.247116959 +0000 UTC m=+66.835770618"
Sep 09 19:55:58.447764 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.447723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:58.654753 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.654717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:58.754242 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:58.754189    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:55:59.218445 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:55:59.218407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:01.571529 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.571444    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"]
Sep 09 19:56:01.600707 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.600667    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"]
Sep 09 19:56:01.600907 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.600816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.603250 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.603224    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 19:56:01.603678 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.603655    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 19:56:01.603804 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.603712    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 19:56:01.604334 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.604297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 19:56:01.604481 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.604462    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 19:56:01.604564 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.604491    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 19:56:01.604564 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.604524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 19:56:01.655684 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.655648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-hub\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.655880 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.655772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.655880 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.655828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.655880 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.655859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-ca\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.656027 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.655886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crzjx\" (UniqueName: \"kubernetes.io/projected/686cb17c-5a77-4142-997e-502dffe12824-kube-api-access-crzjx\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.656027 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.655939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/686cb17c-5a77-4142-997e-502dffe12824-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.745967 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.745933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:01.757074 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.757045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-ca\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.757074 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.757080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crzjx\" (UniqueName: \"kubernetes.io/projected/686cb17c-5a77-4142-997e-502dffe12824-kube-api-access-crzjx\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.757280 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.757110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/686cb17c-5a77-4142-997e-502dffe12824-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.757280 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.757126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-hub\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.757280 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.757188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.757280 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.757228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.757913 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.757880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/686cb17c-5a77-4142-997e-502dffe12824-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.760178 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.760152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-ca\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.760345 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.760294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.760451 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.760403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.760521 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.760487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/686cb17c-5a77-4142-997e-502dffe12824-hub\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.780932 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.780895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crzjx\" (UniqueName: \"kubernetes.io/projected/686cb17c-5a77-4142-997e-502dffe12824-kube-api-access-crzjx\") pod \"cluster-proxy-proxy-agent-8698b758b6-jq4lx\" (UID: \"686cb17c-5a77-4142-997e-502dffe12824\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.829584 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.829496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:01.923247 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.923216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"
Sep 09 19:56:01.930177 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:01.930144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:02.081718 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:02.081633    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:02.174341 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:02.174294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:02.240524 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:02.240486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7mhh" event={"ID":"bc10e010-ba02-48e7-acec-96938127af83","Type":"ContainerStarted","Data":"b5346939f32174fe0e5a2b03f5e89f1e24a3580e47313603b2980ed526307d56"}
Sep 09 19:56:02.240685 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:02.240613    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:56:02.242533 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:02.242510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82kmf" event={"ID":"eb6371e1-d79c-449d-a6a6-1db356e8fdf5","Type":"ContainerStarted","Data":"85aa434ca03b3860e351785443e31335a75ea1e7f517bdbb84f66187d49fdcf5"}
Sep 09 19:56:02.244188 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:02.244169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8r42k" event={"ID":"7bb2809e-c82b-4a2b-86e6-dc341e864dfd","Type":"ContainerStarted","Data":"6c6962dd6c4fed95c15bb1236bd179daaedacaa2298ebdf77e04d63de9d86ab6"}
Sep 09 19:56:02.263158 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:02.262960    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t7mhh" podStartSLOduration=67.500621985 podStartE2EDuration="1m11.262930286s" totalImagesPullingTime="3.762308301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:02.262144554 +0000 UTC m=+70.850798213" watchObservedRunningTime="2026-09-09 19:56:02.262930286 +0000 UTC m=+70.851583943"
Sep 09 19:56:02.481713 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:02.481678    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:02.548023 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:02.547985    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:03.017558 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.017370    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx"]
Sep 09 19:56:03.020018 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:56:03.019993    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod686cb17c_5a77_4142_997e_502dffe12824.slice/crio-dd3af3f266535ea1421cc029eee2b55d59b22183021ca5f071c40480e4719b3a WatchSource:0}: Error finding container dd3af3f266535ea1421cc029eee2b55d59b22183021ca5f071c40480e4719b3a: Status 404 returned error can't find the container with id dd3af3f266535ea1421cc029eee2b55d59b22183021ca5f071c40480e4719b3a
Sep 09 19:56:03.249533 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.249493    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82kmf" event={"ID":"eb6371e1-d79c-449d-a6a6-1db356e8fdf5","Type":"ContainerStarted","Data":"7722e9893fff4ba7118e347bedb3ce15e653b00cdf51dd467da870d0dedfe50f"}
Sep 09 19:56:03.249726 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.249612    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-82kmf"
Sep 09 19:56:03.251452 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.251419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8r42k" event={"ID":"7bb2809e-c82b-4a2b-86e6-dc341e864dfd","Type":"ContainerStarted","Data":"1789d8256cb04e0a89233e8c356d12ea3ece9d421f7e3e7707e5e9d054e5e7a2"}
Sep 09 19:56:03.253017 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.252991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2458j" event={"ID":"34db695b-0f6e-43c2-b92f-65214c6f28e4","Type":"ContainerStarted","Data":"523c5bb10a3a977ab6b3e19abad3306417a74905ee3a993660ff473eeaa6efad"}
Sep 09 19:56:03.254330 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.254288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx" event={"ID":"686cb17c-5a77-4142-997e-502dffe12824","Type":"ContainerStarted","Data":"dd3af3f266535ea1421cc029eee2b55d59b22183021ca5f071c40480e4719b3a"}
Sep 09 19:56:03.269339 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.269246    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-82kmf" podStartSLOduration=35.526523244 podStartE2EDuration="39.269232441s" totalImagesPullingTime="3.742709197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:03.268047086 +0000 UTC m=+71.856700767" watchObservedRunningTime="2026-09-09 19:56:03.269232441 +0000 UTC m=+71.857886097"
Sep 09 19:56:03.284261 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.284212    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8r42k" podStartSLOduration=67.540010371 podStartE2EDuration="1m11.284187032s" totalImagesPullingTime="3.744176661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:03.283948049 +0000 UTC m=+71.872601706" watchObservedRunningTime="2026-09-09 19:56:03.284187032 +0000 UTC m=+71.872840688"
Sep 09 19:56:03.300591 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.300554    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2458j" podStartSLOduration=65.636347698 podStartE2EDuration="1m11.300541759s" totalImagesPullingTime="5.664194061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:03.299896227 +0000 UTC m=+71.888549883" watchObservedRunningTime="2026-09-09 19:56:03.300541759 +0000 UTC m=+71.889195452"
Sep 09 19:56:03.412782 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.412749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:03.480299 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:03.480261    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:04.307980 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:04.307946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:04.409042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:04.409001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:06.885157 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:06.885134    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:06.985726 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:06.985692    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:07.274680 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:07.274646    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx" event={"ID":"686cb17c-5a77-4142-997e-502dffe12824","Type":"ContainerStarted","Data":"395bac59345a513682ca203aff04efeede591e89019b4b584bb99c37c1724d74"}
Sep 09 19:56:07.274680 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:07.274688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx" event={"ID":"686cb17c-5a77-4142-997e-502dffe12824","Type":"ContainerStarted","Data":"cc8ce01a8045da95141b0c0613d0b21a5610fd555de89590ff4e8baf4628c3fd"}
Sep 09 19:56:07.274887 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:07.274700    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx" event={"ID":"686cb17c-5a77-4142-997e-502dffe12824","Type":"ContainerStarted","Data":"b65b4e50deb331f263eb4add9acf0b25b59b6adaafce87214b4756710de880da"}
Sep 09 19:56:07.298017 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:07.297970    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8698b758b6-jq4lx" podStartSLOduration=2.531779561 podStartE2EDuration="6.297955398s" totalImagesPullingTime="3.766175837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:07.296923576 +0000 UTC m=+75.885577234" watchObservedRunningTime="2026-09-09 19:56:07.297955398 +0000 UTC m=+75.886609055"
Sep 09 19:56:09.461326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:09.461286    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:09.524477 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:09.524443    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:10.776094 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:10.776056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:10.836459 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:10.836430    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:11.499286 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:11.499252    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:11.560222 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:11.560188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:12.020771 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:12.020733    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:12.122827 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:12.122783    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:12.199426 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:12.199391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:12.303403 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:12.303286    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:13.261107 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:13.261078    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-82kmf"
Sep 09 19:56:13.278067 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:13.278018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:13.375146 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:13.375110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:13.436537 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:13.436481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:15.755168 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:15.755057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:15.855133 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:15.855104    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:15.953667 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:15.953635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:17.077027 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:17.076990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:17.143633 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:17.143604    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:18.173858 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:18.173820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:19.230490 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:19.230460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:56:19.359905 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:19.359835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:19.504714 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:19.504634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:19.617355 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:19.617297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:20.364704 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.364672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:20.631667 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.631570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:20.707023 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.706992    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9wttd"]
Sep 09 19:56:20.709818 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.709799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.712053 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.712033    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 19:56:20.712444 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.712408    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 19:56:20.712557 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.712461    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 19:56:20.713552 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.713534    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 19:56:20.713651 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.713548    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 19:56:20.713651 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.713587    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 19:56:20.713762 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.713548    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9mskg\""
Sep 09 19:56:20.738104 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.738060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:20.805843 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.805809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cfa4555-ad79-42c5-820f-71d642689608-root\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.805843 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.805847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2x7l\" (UniqueName: \"kubernetes.io/projected/7cfa4555-ad79-42c5-820f-71d642689608-kube-api-access-r2x7l\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.806082 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.805873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-textfile\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.806082 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.805898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.806082 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.805927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.806082 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.805952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-tls\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.806232 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.806077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cfa4555-ad79-42c5-820f-71d642689608-sys\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.806232 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.806128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa4555-ad79-42c5-820f-71d642689608-metrics-client-ca\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.806232 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.806161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-wtmp\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.829760 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.829707    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:20.907326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cfa4555-ad79-42c5-820f-71d642689608-root\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2x7l\" (UniqueName: \"kubernetes.io/projected/7cfa4555-ad79-42c5-820f-71d642689608-kube-api-access-r2x7l\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907326 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-textfile\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907605 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907605 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7cfa4555-ad79-42c5-820f-71d642689608-root\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907605 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907605 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-tls\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907605 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cfa4555-ad79-42c5-820f-71d642689608-sys\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907605 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa4555-ad79-42c5-820f-71d642689608-metrics-client-ca\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907605 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-wtmp\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907921 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cfa4555-ad79-42c5-820f-71d642689608-sys\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907921 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-wtmp\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907921 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907691    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-textfile\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.907921 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:56:20.907780    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 19:56:20.907921 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:56:20.907878    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-tls podName:7cfa4555-ad79-42c5-820f-71d642689608 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:21.407857092 +0000 UTC m=+89.996510731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-tls") pod "node-exporter-9wttd" (UID: "7cfa4555-ad79-42c5-820f-71d642689608") : secret "node-exporter-tls" not found
Sep 09 19:56:20.908163 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.907925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.908163 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.908074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa4555-ad79-42c5-820f-71d642689608-metrics-client-ca\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.909842 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.909820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.916603 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.916582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2x7l\" (UniqueName: \"kubernetes.io/projected/7cfa4555-ad79-42c5-820f-71d642689608-kube-api-access-r2x7l\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:20.932846 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:20.932819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.030302 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:21.030266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.127534 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:21.127502    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.258183 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:21.258106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.412256 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:21.412226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-tls\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:21.414723 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:21.414692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7cfa4555-ad79-42c5-820f-71d642689608-node-exporter-tls\") pod \"node-exporter-9wttd\" (UID: \"7cfa4555-ad79-42c5-820f-71d642689608\") " pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:21.603297 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:21.603253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.619756 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:21.619727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wttd"
Sep 09 19:56:21.630432 ip-10-0-138-141 kubenswrapper[2578]: W0909 19:56:21.630399    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfa4555_ad79_42c5_820f_71d642689608.slice/crio-8bf8f9dc3396d4800949030c18f27424b1037367b1a76263fb3e781087571c70 WatchSource:0}: Error finding container 8bf8f9dc3396d4800949030c18f27424b1037367b1a76263fb3e781087571c70: Status 404 returned error can't find the container with id 8bf8f9dc3396d4800949030c18f27424b1037367b1a76263fb3e781087571c70
Sep 09 19:56:21.721048 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:21.721012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:21.848963 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:21.848933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:22.025992 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:22.025896    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:22.334592 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:22.334518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wttd" event={"ID":"7cfa4555-ad79-42c5-820f-71d642689608","Type":"ContainerStarted","Data":"8bf8f9dc3396d4800949030c18f27424b1037367b1a76263fb3e781087571c70"}
Sep 09 19:56:22.608480 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:22.608426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:22.716738 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:22.716708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:23.339670 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:23.339630    2578 generic.go:356] "Generic (PLEG): container finished" podID="7cfa4555-ad79-42c5-820f-71d642689608" containerID="61be7b855f3a666157189e8449a7ab14ef32a2cd0f59790e8f84a11b8478a514" exitCode=0
Sep 09 19:56:23.339835 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:23.339701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wttd" event={"ID":"7cfa4555-ad79-42c5-820f-71d642689608","Type":"ContainerDied","Data":"61be7b855f3a666157189e8449a7ab14ef32a2cd0f59790e8f84a11b8478a514"}
Sep 09 19:56:23.914891 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:23.914857    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:24.237911 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:24.237829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:24.319704 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:24.319670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:24.345073 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:24.345035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wttd" event={"ID":"7cfa4555-ad79-42c5-820f-71d642689608","Type":"ContainerStarted","Data":"022883e8f9582e6aceee1f96e167ca87258b47fa495c982ef712179a1e8bf291"}
Sep 09 19:56:24.345236 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:24.345096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wttd" event={"ID":"7cfa4555-ad79-42c5-820f-71d642689608","Type":"ContainerStarted","Data":"6180bbecddf135db7875bc74983c8e283c053d9e467a62e64ef5f5142560e417"}
Sep 09 19:56:24.349464 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:24.349440    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bc687dd7-kf6hm"]
Sep 09 19:56:24.372780 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:24.372725    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9wttd" podStartSLOduration=2.8079621230000003 podStartE2EDuration="4.372707159s" totalImagesPullingTime="1.501319457s" totalInitContainerRuntime="63.425579ms" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:24.371841339 +0000 UTC m=+92.960495019" watchObservedRunningTime="2026-09-09 19:56:24.372707159 +0000 UTC m=+92.961360849"
Sep 09 19:56:24.453931 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:24.453897    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:26.153255 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:26.153217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:26.250057 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:26.250017    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:27.011819 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:27.011787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:28.111275 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:28.111187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:28.211017 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:28.210982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:31.456753 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:31.456714    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:33.257532 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:33.257496    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t7mhh"
Sep 09 19:56:34.310392 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:34.310361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:34.431839 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:34.431800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:37.739014 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:37.738982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:37.846137 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:37.846088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:37.912563 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:37.912528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:40.174177 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:40.174135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:40.236983 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:40.236950    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:43.412378 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:43.412261    2578 generic.go:356] "Generic (PLEG): container finished" podID="8fcff86d-6410-409d-975b-877aa305ceb1" containerID="3474d15f72c8246e17d65286af649dc6c6756a5332ccd14ffdd6e1ed81f20e01" exitCode=0
Sep 09 19:56:43.412378 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:43.412332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q" event={"ID":"8fcff86d-6410-409d-975b-877aa305ceb1","Type":"ContainerDied","Data":"3474d15f72c8246e17d65286af649dc6c6756a5332ccd14ffdd6e1ed81f20e01"}
Sep 09 19:56:43.412939 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:43.412747    2578 scope.go:117] "RemoveContainer" containerID="3474d15f72c8246e17d65286af649dc6c6756a5332ccd14ffdd6e1ed81f20e01"
Sep 09 19:56:44.223349 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:44.223319    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5x4l5_65266674-4ef3-4abf-b20c-3c06447af305/serve-healthcheck-canary/0.log"
Sep 09 19:56:44.417908 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:44.417833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-s2v7q" event={"ID":"8fcff86d-6410-409d-975b-877aa305ceb1","Type":"ContainerStarted","Data":"47b3baa17ef2c3b82af0ab5db21457e90d4bcfac682afe0a55b19beeb094b479"}
Sep 09 19:56:49.333011 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.332974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:49.373064 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.373017    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm" podUID="76a619ca-d897-4baa-bf31-d0aed2ca4ffd" containerName="registry" containerID="cri-o://784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93" gracePeriod=30
Sep 09 19:56:49.433292 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.433257    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:56:49.604586 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.604562    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:56:49.724945 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.724911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-ca-trust-extracted\") pod \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") "
Sep 09 19:56:49.725110 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.724960    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-image-registry-private-configuration\") pod \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") "
Sep 09 19:56:49.725110 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.724988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-installation-pull-secrets\") pod \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") "
Sep 09 19:56:49.725201 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.725164    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") pod \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") "
Sep 09 19:56:49.725255 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.725214    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-bound-sa-token\") pod \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") "
Sep 09 19:56:49.725341 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.725262    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-certificates\") pod \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") "
Sep 09 19:56:49.725341 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.725331    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-trusted-ca\") pod \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") "
Sep 09 19:56:49.725448 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.725371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2zrj\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-kube-api-access-p2zrj\") pod \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\" (UID: \"76a619ca-d897-4baa-bf31-d0aed2ca4ffd\") "
Sep 09 19:56:49.725934 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.725855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "76a619ca-d897-4baa-bf31-d0aed2ca4ffd" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:56:49.725934 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.725895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "76a619ca-d897-4baa-bf31-d0aed2ca4ffd" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:56:49.727919 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.727867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "76a619ca-d897-4baa-bf31-d0aed2ca4ffd" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:56:49.728038 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.727972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "76a619ca-d897-4baa-bf31-d0aed2ca4ffd" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:56:49.728103 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.728021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "76a619ca-d897-4baa-bf31-d0aed2ca4ffd" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:56:49.728103 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.728085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "76a619ca-d897-4baa-bf31-d0aed2ca4ffd" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:56:49.728208 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.728161    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-kube-api-access-p2zrj" (OuterVolumeSpecName: "kube-api-access-p2zrj") pod "76a619ca-d897-4baa-bf31-d0aed2ca4ffd" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd"). InnerVolumeSpecName "kube-api-access-p2zrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:56:49.733839 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.733812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "76a619ca-d897-4baa-bf31-d0aed2ca4ffd" (UID: "76a619ca-d897-4baa-bf31-d0aed2ca4ffd"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:56:49.826820 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.826778    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-bound-sa-token\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.826820 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.826814    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-certificates\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.826820 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.826826    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-trusted-ca\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.827049 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.826834    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2zrj\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-kube-api-access-p2zrj\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.827049 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.826842    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-ca-trust-extracted\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.827049 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.826851    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-image-registry-private-configuration\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.827049 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.826862    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-installation-pull-secrets\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 19:56:49.827049 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:49.826870    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/76a619ca-d897-4baa-bf31-d0aed2ca4ffd-registry-tls\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 19:56:50.439564 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:50.439532    2578 generic.go:356] "Generic (PLEG): container finished" podID="76a619ca-d897-4baa-bf31-d0aed2ca4ffd" containerID="784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93" exitCode=0
Sep 09 19:56:50.439963 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:50.439572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm" event={"ID":"76a619ca-d897-4baa-bf31-d0aed2ca4ffd","Type":"ContainerDied","Data":"784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93"}
Sep 09 19:56:50.439963 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:50.439576    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm"
Sep 09 19:56:50.439963 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:50.439614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bc687dd7-kf6hm" event={"ID":"76a619ca-d897-4baa-bf31-d0aed2ca4ffd","Type":"ContainerDied","Data":"d9ce9227b9342dc303444a158196d38ab8701ca85fc39c0d10b9c1d28c2158dd"}
Sep 09 19:56:50.439963 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:50.439634    2578 scope.go:117] "RemoveContainer" containerID="784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93"
Sep 09 19:56:50.447827 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:50.447807    2578 scope.go:117] "RemoveContainer" containerID="784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93"
Sep 09 19:56:50.448100 ip-10-0-138-141 kubenswrapper[2578]: E0909 19:56:50.448078    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93\": container with ID starting with 784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93 not found: ID does not exist" containerID="784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93"
Sep 09 19:56:50.448156 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:50.448109    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93"} err="failed to get container status \"784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93\": rpc error: code = NotFound desc = could not find container \"784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93\": container with ID starting with 784ac65192c572d3b5559865180ce203717abf9ea71b107a84bd06fc20928c93 not found: ID does not exist"
Sep 09 19:56:50.457179 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:50.457155    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bc687dd7-kf6hm"]
Sep 09 19:56:50.463134 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:50.463110    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55bc687dd7-kf6hm"]
Sep 09 19:56:51.895690 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:51.895650    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76a619ca-d897-4baa-bf31-d0aed2ca4ffd" path="/var/lib/kubelet/pods/76a619ca-d897-4baa-bf31-d0aed2ca4ffd/volumes"
Sep 09 19:56:54.705478 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:56:54.705440    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:02.486901 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:02.486808    2578 generic.go:356] "Generic (PLEG): container finished" podID="82c99ae4-950c-4770-bd66-d8778fe3baf4" containerID="862e2bb1db6a54ca0400d8b27002bd23e55f95810b51b035b1d3701efc59c1ef" exitCode=0
Sep 09 19:57:02.486901 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:02.486885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g" event={"ID":"82c99ae4-950c-4770-bd66-d8778fe3baf4","Type":"ContainerDied","Data":"862e2bb1db6a54ca0400d8b27002bd23e55f95810b51b035b1d3701efc59c1ef"}
Sep 09 19:57:02.487278 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:02.487233    2578 scope.go:117] "RemoveContainer" containerID="862e2bb1db6a54ca0400d8b27002bd23e55f95810b51b035b1d3701efc59c1ef"
Sep 09 19:57:03.491774 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:03.491734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-9ts9g" event={"ID":"82c99ae4-950c-4770-bd66-d8778fe3baf4","Type":"ContainerStarted","Data":"c478cafd55902281278557846353b0956e24b8a4cc7b9daf1a19869d2df0f25b"}
Sep 09 19:57:07.506642 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:07.506608    2578 generic.go:356] "Generic (PLEG): container finished" podID="08238358-ba3a-492a-b867-8edd7905ad48" containerID="1ff96cad60d95dc5f4ba6c1a30642f6ed02969d314664ce125049f7f6500ff6f" exitCode=0
Sep 09 19:57:07.507014 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:07.506666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx" event={"ID":"08238358-ba3a-492a-b867-8edd7905ad48","Type":"ContainerDied","Data":"1ff96cad60d95dc5f4ba6c1a30642f6ed02969d314664ce125049f7f6500ff6f"}
Sep 09 19:57:07.507014 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:07.506988    2578 scope.go:117] "RemoveContainer" containerID="1ff96cad60d95dc5f4ba6c1a30642f6ed02969d314664ce125049f7f6500ff6f"
Sep 09 19:57:08.512074 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:08.512034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6p9hx" event={"ID":"08238358-ba3a-492a-b867-8edd7905ad48","Type":"ContainerStarted","Data":"c324875d86fdcdafac19aeed8e1e0f7b4c23896f37969bfc352e2be57489b93f"}
Sep 09 19:57:12.546998 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:12.546967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:12.646996 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:12.646961    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:12.710674 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:12.710646    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:39.966519 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:39.966478    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:40.207764 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:40.207723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:49.116672 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:49.116639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:57:49.177923 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:57:49.177885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:00.106632 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:58:00.106593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:00.207370 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:58:00.207335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:00.271141 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:58:00.271113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:14.410393 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:58:14.410357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:55.226098 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:58:55.226055    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:55.282009 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:58:55.281975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:56.925042 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:58:56.924998    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:58:56.984703 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:58:56.984673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:07.173790 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:07.173749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:07.233348 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:07.233300    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:07.403277 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:07.403243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:07.505072 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:07.504990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:07.561499 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:07.561468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:19.323148 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:19.323070    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:19.423925 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:19.423891    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:30.273174 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:30.273110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:30.335514 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:30.335479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:32.259514 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:32.259474    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 19:59:51.803387 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:51.803355    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 19:59:51.808126 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:51.808106    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 19:59:51.811446 ip-10-0-138-141 kubenswrapper[2578]: I0909 19:59:51.811417    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:00:23.589225 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:00:23.589175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:00:33.271353 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:00:33.271274    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:00:33.369210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:00:33.369171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:00:33.612609 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:00:33.612577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:01.145917 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:01.145882    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:01.258407 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:01.258375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:07.974481 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:07.974435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:08.269271 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.269187    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-dxrh8"]
Sep 09 20:01:08.269608 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.269549    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76a619ca-d897-4baa-bf31-d0aed2ca4ffd" containerName="registry"
Sep 09 20:01:08.269608 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.269571    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76a619ca-d897-4baa-bf31-d0aed2ca4ffd" containerName="registry"
Sep 09 20:01:08.269760 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.269647    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76a619ca-d897-4baa-bf31-d0aed2ca4ffd" containerName="registry"
Sep 09 20:01:08.271744 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.271724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.274206 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.274188    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:01:08.274333 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.274237    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:01:08.274469 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.274453    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:01:08.274680 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.274665    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:01:08.275089 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.275074    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:01:08.275479 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.275458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zbjxt\""
Sep 09 20:01:08.285413 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.285392    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-dxrh8"]
Sep 09 20:01:08.376521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.376491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-certificates\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.376521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.376521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp286\" (UniqueName: \"kubernetes.io/projected/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-kube-api-access-xp286\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.376716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.376555    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-cabundle0\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.477557 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.477523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-certificates\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.477557 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.477558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp286\" (UniqueName: \"kubernetes.io/projected/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-kube-api-access-xp286\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.477784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.477591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-cabundle0\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.477784 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:01:08.477672    2578 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:01:08.477784 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:01:08.477691    2578 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:01:08.477784 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:01:08.477701    2578 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-dxrh8: references non-existent secret key: ca.crt
Sep 09 20:01:08.477784 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:01:08.477759    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-certificates podName:a82a418e-e2b7-4dbe-ba05-bb67f176f2f5 nodeName:}" failed. No retries permitted until 2026-09-09 20:01:08.977743884 +0000 UTC m=+377.566397519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-certificates") pod "keda-operator-579c8958f7-dxrh8" (UID: "a82a418e-e2b7-4dbe-ba05-bb67f176f2f5") : references non-existent secret key: ca.crt
Sep 09 20:01:08.478223 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.478204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-cabundle0\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.489706 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.489681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp286\" (UniqueName: \"kubernetes.io/projected/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-kube-api-access-xp286\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.981602 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.981564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-certificates\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:08.983898 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:08.983863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a82a418e-e2b7-4dbe-ba05-bb67f176f2f5-certificates\") pod \"keda-operator-579c8958f7-dxrh8\" (UID: \"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5\") " pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:09.182411 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:09.182372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:09.456417 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:09.456388    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-dxrh8"]
Sep 09 20:01:09.459772 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:01:09.459746    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82a418e_e2b7_4dbe_ba05_bb67f176f2f5.slice/crio-8d414aa6388edcd759876cb87b19689a3da9bc4108d34a608cdc3961777ebc23 WatchSource:0}: Error finding container 8d414aa6388edcd759876cb87b19689a3da9bc4108d34a608cdc3961777ebc23: Status 404 returned error can't find the container with id 8d414aa6388edcd759876cb87b19689a3da9bc4108d34a608cdc3961777ebc23
Sep 09 20:01:09.461497 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:09.461480    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:01:09.742142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:09.742102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:09.838496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:09.838463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:10.355451 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:10.355414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-dxrh8" event={"ID":"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5","Type":"ContainerStarted","Data":"8d414aa6388edcd759876cb87b19689a3da9bc4108d34a608cdc3961777ebc23"}
Sep 09 20:01:10.691836 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:10.691741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:10.857121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:10.857082    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:13.369550 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:13.369510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-dxrh8" event={"ID":"a82a418e-e2b7-4dbe-ba05-bb67f176f2f5","Type":"ContainerStarted","Data":"9bed096a36c20244821fdbca0fa5f5d94a1574e60dfe1fb0ec069d5dd3c4670a"}
Sep 09 20:01:13.369899 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:13.369605    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:13.388790 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:13.388738    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-dxrh8" podStartSLOduration=1.5612841579999999 podStartE2EDuration="5.38871987s" totalImagesPullingTime="3.827435712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:13.388176207 +0000 UTC m=+381.976829865" watchObservedRunningTime="2026-09-09 20:01:13.38871987 +0000 UTC m=+381.977373526"
Sep 09 20:01:14.867961 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:14.867932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:14.968747 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:14.968717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:22.522998 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:22.522961    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:23.461608 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:23.461576    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:23.554721 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:23.554689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:33.038761 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:33.038717    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:33.138388 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:33.138356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:33.238603 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:33.238564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:01:34.375825 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:34.375791    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-dxrh8"
Sep 09 20:01:59.910729 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:01:59.910688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:00.131020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:00.130985    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:11.148121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:11.148083    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:11.917457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:11.917422    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:13.367619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.367587    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"]
Sep 09 20:02:13.369863 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.369839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"
Sep 09 20:02:13.375429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.375404    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 20:02:13.375534 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.375465    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kcnnr\""
Sep 09 20:02:13.375534 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.375472    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:02:13.389031 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.388998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"]
Sep 09 20:02:13.452839 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.452806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdpwf\" (UniqueName: \"kubernetes.io/projected/766c8733-fb15-4395-8f44-a7a84eee100a-kube-api-access-hdpwf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qrj4g\" (UID: \"766c8733-fb15-4395-8f44-a7a84eee100a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"
Sep 09 20:02:13.453012 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.452873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/766c8733-fb15-4395-8f44-a7a84eee100a-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qrj4g\" (UID: \"766c8733-fb15-4395-8f44-a7a84eee100a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"
Sep 09 20:02:13.553855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.553817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/766c8733-fb15-4395-8f44-a7a84eee100a-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qrj4g\" (UID: \"766c8733-fb15-4395-8f44-a7a84eee100a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"
Sep 09 20:02:13.553855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.553858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdpwf\" (UniqueName: \"kubernetes.io/projected/766c8733-fb15-4395-8f44-a7a84eee100a-kube-api-access-hdpwf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qrj4g\" (UID: \"766c8733-fb15-4395-8f44-a7a84eee100a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"
Sep 09 20:02:13.554229 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.554209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/766c8733-fb15-4395-8f44-a7a84eee100a-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qrj4g\" (UID: \"766c8733-fb15-4395-8f44-a7a84eee100a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"
Sep 09 20:02:13.565794 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.565770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdpwf\" (UniqueName: \"kubernetes.io/projected/766c8733-fb15-4395-8f44-a7a84eee100a-kube-api-access-hdpwf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-qrj4g\" (UID: \"766c8733-fb15-4395-8f44-a7a84eee100a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"
Sep 09 20:02:13.678780 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.678687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"
Sep 09 20:02:13.960181 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:13.960106    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g"]
Sep 09 20:02:13.963988 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:02:13.963962    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766c8733_fb15_4395_8f44_a7a84eee100a.slice/crio-008348a522255b612b10907fd862cb932fcc2f23e67b65a488ea2fd4d2fb794e WatchSource:0}: Error finding container 008348a522255b612b10907fd862cb932fcc2f23e67b65a488ea2fd4d2fb794e: Status 404 returned error can't find the container with id 008348a522255b612b10907fd862cb932fcc2f23e67b65a488ea2fd4d2fb794e
Sep 09 20:02:14.584470 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:14.584436    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g" event={"ID":"766c8733-fb15-4395-8f44-a7a84eee100a","Type":"ContainerStarted","Data":"008348a522255b612b10907fd862cb932fcc2f23e67b65a488ea2fd4d2fb794e"}
Sep 09 20:02:15.021423 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:15.021389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:15.135470 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:15.135435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:19.608118 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:19.608078    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g" event={"ID":"766c8733-fb15-4395-8f44-a7a84eee100a","Type":"ContainerStarted","Data":"ce10e7a2ee67820eae0786005d5e84bcf5591c5440ade418de19fb8263eac552"}
Sep 09 20:02:19.630833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:19.630784    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-qrj4g" podStartSLOduration=1.471342391 podStartE2EDuration="6.630767867s" totalImagesPullingTime="5.159425476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:19.629328394 +0000 UTC m=+448.217982050" watchObservedRunningTime="2026-09-09 20:02:19.630767867 +0000 UTC m=+448.219421524"
Sep 09 20:02:20.748716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:20.748680    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:20.882151 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:20.882114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:21.622271 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:21.622240    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:21.762130 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:21.762097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:21.881063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:21.880969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:22.390240 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:22.390204    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:22.994940 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:22.994903    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"]
Sep 09 20:02:22.997433 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:22.997415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"
Sep 09 20:02:23.001967 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.001945    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 20:02:23.002127 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.002110    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 20:02:23.002182 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.002159    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-n2rn8\""
Sep 09 20:02:23.019463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.019423    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"]
Sep 09 20:02:23.124765 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.124722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f88e112-357e-41f8-b750-a145af84b788-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mnr2q\" (UID: \"0f88e112-357e-41f8-b750-a145af84b788\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"
Sep 09 20:02:23.124961 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.124779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qb5z\" (UniqueName: \"kubernetes.io/projected/0f88e112-357e-41f8-b750-a145af84b788-kube-api-access-4qb5z\") pod \"cert-manager-cainjector-559cdd8c87-mnr2q\" (UID: \"0f88e112-357e-41f8-b750-a145af84b788\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"
Sep 09 20:02:23.226104 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.226050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f88e112-357e-41f8-b750-a145af84b788-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mnr2q\" (UID: \"0f88e112-357e-41f8-b750-a145af84b788\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"
Sep 09 20:02:23.226104 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.226115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qb5z\" (UniqueName: \"kubernetes.io/projected/0f88e112-357e-41f8-b750-a145af84b788-kube-api-access-4qb5z\") pod \"cert-manager-cainjector-559cdd8c87-mnr2q\" (UID: \"0f88e112-357e-41f8-b750-a145af84b788\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"
Sep 09 20:02:23.236643 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.236610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f88e112-357e-41f8-b750-a145af84b788-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mnr2q\" (UID: \"0f88e112-357e-41f8-b750-a145af84b788\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"
Sep 09 20:02:23.236783 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.236747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qb5z\" (UniqueName: \"kubernetes.io/projected/0f88e112-357e-41f8-b750-a145af84b788-kube-api-access-4qb5z\") pod \"cert-manager-cainjector-559cdd8c87-mnr2q\" (UID: \"0f88e112-357e-41f8-b750-a145af84b788\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"
Sep 09 20:02:23.306429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.306330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"
Sep 09 20:02:23.587009 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.586924    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q"]
Sep 09 20:02:23.589955 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:02:23.589927    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f88e112_357e_41f8_b750_a145af84b788.slice/crio-f5af705ec131760debb2c28cc1e4912076a598ee715eab3c5f5ef97f8b49cd5b WatchSource:0}: Error finding container f5af705ec131760debb2c28cc1e4912076a598ee715eab3c5f5ef97f8b49cd5b: Status 404 returned error can't find the container with id f5af705ec131760debb2c28cc1e4912076a598ee715eab3c5f5ef97f8b49cd5b
Sep 09 20:02:23.625475 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:23.625443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q" event={"ID":"0f88e112-357e-41f8-b750-a145af84b788","Type":"ContainerStarted","Data":"f5af705ec131760debb2c28cc1e4912076a598ee715eab3c5f5ef97f8b49cd5b"}
Sep 09 20:02:26.642297 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:26.642248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q" event={"ID":"0f88e112-357e-41f8-b750-a145af84b788","Type":"ContainerStarted","Data":"fb1216caa32b3808cf2e394249ede030dbaed1747c826dcfac1c7919970fd9dd"}
Sep 09 20:02:26.659112 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:26.659052    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mnr2q" podStartSLOduration=2.241301484 podStartE2EDuration="4.659037571s" totalImagesPullingTime="2.417736087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:26.657577327 +0000 UTC m=+455.246230983" watchObservedRunningTime="2026-09-09 20:02:26.659037571 +0000 UTC m=+455.247691228"
Sep 09 20:02:26.793043 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:26.793009    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:26.894543 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:26.894441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:27.353035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:27.353008    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:27.679948 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:27.679865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:29.507417 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:29.507377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:29.568717 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:29.568688    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:30.516991 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:30.516959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:34.532598 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:34.532564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:34.634781 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:34.634745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:45.193366 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:45.193325    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:49.964415 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:49.964383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:50.074411 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:50.074367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:51.193777 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:51.193739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:54.074210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:54.074174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:54.218953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:54.218919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:58.388507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.388474    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"]
Sep 09 20:02:58.391193 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.391167    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.406803 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.406771    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:02:58.406944 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.406814    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-52pdr\""
Sep 09 20:02:58.406944 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.406771    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:02:58.407154 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.407137    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:02:58.420475 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.420451    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"]
Sep 09 20:02:58.496004 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.495966    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870011d5-3b7b-4211-8007-9f917ffa1944-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.496004 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.496004    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870011d5-3b7b-4211-8007-9f917ffa1944-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.496211 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.496047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.496211 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.496081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.496211 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.496101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.496211 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.496151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.496211 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.496185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhms\" (UniqueName: \"kubernetes.io/projected/870011d5-3b7b-4211-8007-9f917ffa1944-kube-api-access-cxhms\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.496211 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.496209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870011d5-3b7b-4211-8007-9f917ffa1944-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.496434 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.496235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.509351 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.509299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:58.597578 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.597543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.597727 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.597586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhms\" (UniqueName: \"kubernetes.io/projected/870011d5-3b7b-4211-8007-9f917ffa1944-kube-api-access-cxhms\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.597727 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.597605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870011d5-3b7b-4211-8007-9f917ffa1944-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.597727 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.597638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.597865 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.597841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870011d5-3b7b-4211-8007-9f917ffa1944-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.597918 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.597894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870011d5-3b7b-4211-8007-9f917ffa1944-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.597968 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.597944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.598079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.597999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.598079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.598054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.598187 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.598118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.598187 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.598151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.598462 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.598438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870011d5-3b7b-4211-8007-9f917ffa1944-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.598569 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.598459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.598569 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.598482    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.599513 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.599487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:02:58.600415 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.600396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870011d5-3b7b-4211-8007-9f917ffa1944-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.600579 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.600561    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870011d5-3b7b-4211-8007-9f917ffa1944-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.607044 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.607020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870011d5-3b7b-4211-8007-9f917ffa1944-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.607181 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.607165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhms\" (UniqueName: \"kubernetes.io/projected/870011d5-3b7b-4211-8007-9f917ffa1944-kube-api-access-cxhms\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv\" (UID: \"870011d5-3b7b-4211-8007-9f917ffa1944\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.700788 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.700701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:02:58.978406 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:58.978318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"]
Sep 09 20:02:58.980838 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:02:58.980811    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870011d5_3b7b_4211_8007_9f917ffa1944.slice/crio-f6f40da635809e5d46f32848ce4f15d0d678a95e5e7cf5435151255414b4a1d0 WatchSource:0}: Error finding container f6f40da635809e5d46f32848ce4f15d0d678a95e5e7cf5435151255414b4a1d0: Status 404 returned error can't find the container with id f6f40da635809e5d46f32848ce4f15d0d678a95e5e7cf5435151255414b4a1d0
Sep 09 20:02:59.777356 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:02:59.777303    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" event={"ID":"870011d5-3b7b-4211-8007-9f917ffa1944","Type":"ContainerStarted","Data":"f6f40da635809e5d46f32848ce4f15d0d678a95e5e7cf5435151255414b4a1d0"}
Sep 09 20:03:01.358952 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:01.358920    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:03.303153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.303120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:03.407664 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.407624    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"]
Sep 09 20:03:03.413325 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.413289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.417547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.417524    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 20:03:03.418492 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.418447    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 20:03:03.418492 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.418463    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 20:03:03.418677 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.418498    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-f48g9\""
Sep 09 20:03:03.418766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.418744    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 20:03:03.418863 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.418772    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:03:03.420858 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.420794    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:03.422820 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.422763    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"]
Sep 09 20:03:03.539155 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.539107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-cert\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.539359 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.539172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-manager-config\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.539359 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.539271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjjxl\" (UniqueName: \"kubernetes.io/projected/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-kube-api-access-mjjxl\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.539359 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.539294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-metrics-cert\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.640108 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.640061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjjxl\" (UniqueName: \"kubernetes.io/projected/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-kube-api-access-mjjxl\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.640301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.640125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-metrics-cert\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.640301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.640200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-cert\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.640301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.640244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-manager-config\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.641098 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.641065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-manager-config\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.642796 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.642776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-cert\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.642880 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.642850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-metrics-cert\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.650633 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.650611    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjjxl\" (UniqueName: \"kubernetes.io/projected/5e80ba5d-4e55-47d6-b516-0dfad1d7223b-kube-api-access-mjjxl\") pod \"lws-controller-manager-7ff45979b5-r9jv5\" (UID: \"5e80ba5d-4e55-47d6-b516-0dfad1d7223b\") " pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.725739 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.725700    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:03.790744 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.790695    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:03:03.790890 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.790805    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:03:03.790890 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:03.790853    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:03:04.026292 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:04.026253    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"]
Sep 09 20:03:04.026751 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:03:04.026724    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e80ba5d_4e55_47d6_b516_0dfad1d7223b.slice/crio-7688c0c534c76df1b304a36f74082f9451c0e2483cf5c5f2526e7e493965dc4b WatchSource:0}: Error finding container 7688c0c534c76df1b304a36f74082f9451c0e2483cf5c5f2526e7e493965dc4b: Status 404 returned error can't find the container with id 7688c0c534c76df1b304a36f74082f9451c0e2483cf5c5f2526e7e493965dc4b
Sep 09 20:03:04.800064 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:04.800023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" event={"ID":"870011d5-3b7b-4211-8007-9f917ffa1944","Type":"ContainerStarted","Data":"0d3912d8e4b99260b9702d66303a303fdddc93c4ca4166e8dc67f6454691bd07"}
Sep 09 20:03:04.801197 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:04.801175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5" event={"ID":"5e80ba5d-4e55-47d6-b516-0dfad1d7223b","Type":"ContainerStarted","Data":"7688c0c534c76df1b304a36f74082f9451c0e2483cf5c5f2526e7e493965dc4b"}
Sep 09 20:03:04.822524 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:04.822469    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" podStartSLOduration=2.014951792 podStartE2EDuration="6.822453202s" totalImagesPullingTime="4.80750141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:04.821734775 +0000 UTC m=+493.410388435" watchObservedRunningTime="2026-09-09 20:03:04.822453202 +0000 UTC m=+493.411106855"
Sep 09 20:03:05.701483 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:05.701446    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:03:05.706666 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:05.706637    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:03:05.805909 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:05.805869    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:03:05.806896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:05.806870    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:03:06.029930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:06.029846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:06.143281 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:06.143248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:06.811509 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:06.811422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5" event={"ID":"5e80ba5d-4e55-47d6-b516-0dfad1d7223b","Type":"ContainerStarted","Data":"449c9af1610a8a1928185076aeae8c8884e439738008e567e8f3c94a2d5c553e"}
Sep 09 20:03:06.811982 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:06.811619    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:06.828898 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:06.828824    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5" podStartSLOduration=1.297478993 podStartE2EDuration="3.82881128s" totalImagesPullingTime="2.531332287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:06.82805853 +0000 UTC m=+495.416712199" watchObservedRunningTime="2026-09-09 20:03:06.82881128 +0000 UTC m=+495.417464936"
Sep 09 20:03:17.818231 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:17.818201    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7ff45979b5-r9jv5"
Sep 09 20:03:27.024493 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:27.024451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:29.421660 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:29.421620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:29.869168 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:29.869136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:30.215222 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:30.215140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:31.076701 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:31.076662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:33.020518 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:33.020481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:33.146716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:33.146664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:33.634011 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:33.633978    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:34.008020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:34.007937    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:34.287520 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:34.287439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:34.391624 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:34.391590    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:36.607976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:36.607938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:38.573161 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:38.573119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:39.625788 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:39.625751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:39.734116 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:39.734078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:41.180703 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.180654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:41.181074 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.180887    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"]
Sep 09 20:03:41.188275 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.188257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.191212 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.191191    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 20:03:41.191434 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.191419    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 20:03:41.191999 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.191984    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zkj67\""
Sep 09 20:03:41.192101 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.192082    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 20:03:41.192101 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.192093    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 20:03:41.199848 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.199812    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"]
Sep 09 20:03:41.242566 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.242534    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.242728 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.242577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.242728 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.242637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk49h\" (UniqueName: \"kubernetes.io/projected/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-kube-api-access-vk49h\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.343054 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.343011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.343222 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.343063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.343222 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.343100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk49h\" (UniqueName: \"kubernetes.io/projected/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-kube-api-access-vk49h\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.343297 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:03:41.343230    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 20:03:41.343364 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:03:41.343343    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-plugin-serving-cert podName:cd920ac6-11e4-4c6a-99c1-d48bbf0500ba nodeName:}" failed. No retries permitted until 2026-09-09 20:03:41.843295587 +0000 UTC m=+530.431949233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-9hp8f" (UID: "cd920ac6-11e4-4c6a-99c1-d48bbf0500ba") : secret "plugin-serving-cert" not found
Sep 09 20:03:41.343723 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.343701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.353108 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.353072    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk49h\" (UniqueName: \"kubernetes.io/projected/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-kube-api-access-vk49h\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.846643 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:41.846606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:41.846829 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:03:41.846724    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 09 20:03:41.846829 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:03:41.846786    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-plugin-serving-cert podName:cd920ac6-11e4-4c6a-99c1-d48bbf0500ba nodeName:}" failed. No retries permitted until 2026-09-09 20:03:42.846772201 +0000 UTC m=+531.435425837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-9hp8f" (UID: "cd920ac6-11e4-4c6a-99c1-d48bbf0500ba") : secret "plugin-serving-cert" not found
Sep 09 20:03:42.166477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:42.166389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:42.853613 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:42.853577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:42.855995 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:42.855970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd920ac6-11e4-4c6a-99c1-d48bbf0500ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-9hp8f\" (UID: \"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:42.997478 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:42.997441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"
Sep 09 20:03:43.266072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:43.265992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f"]
Sep 09 20:03:43.269144 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:03:43.269109    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd920ac6_11e4_4c6a_99c1_d48bbf0500ba.slice/crio-1e10b7d03ec56466a0c2ffd7e86ffd34f9068f5cb82c20b444d23c1f0f135fc1 WatchSource:0}: Error finding container 1e10b7d03ec56466a0c2ffd7e86ffd34f9068f5cb82c20b444d23c1f0f135fc1: Status 404 returned error can't find the container with id 1e10b7d03ec56466a0c2ffd7e86ffd34f9068f5cb82c20b444d23c1f0f135fc1
Sep 09 20:03:43.962982 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:43.962949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f" event={"ID":"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba","Type":"ContainerStarted","Data":"1e10b7d03ec56466a0c2ffd7e86ffd34f9068f5cb82c20b444d23c1f0f135fc1"}
Sep 09 20:03:44.242193 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:44.242096    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:44.339922 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:44.339836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:44.439304 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:44.439249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:44.540522 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:44.540259    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:44.640805 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:44.640742    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:44.709673 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:44.709615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:48.988929 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:48.988895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f" event={"ID":"cd920ac6-11e4-4c6a-99c1-d48bbf0500ba","Type":"ContainerStarted","Data":"1d1cd34c2be143cca86715db229a8eaf4b249b3fc288767d50c6b0d8388e58f8"}
Sep 09 20:03:49.016094 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:49.016040    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-9hp8f" podStartSLOduration=3.281100211 podStartE2EDuration="8.01602568s" totalImagesPullingTime="4.734925469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:49.013979319 +0000 UTC m=+537.602632976" watchObservedRunningTime="2026-09-09 20:03:49.01602568 +0000 UTC m=+537.604679337"
Sep 09 20:03:53.658737 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:53.658695    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:53.778739 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:53.778704    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:54.248070 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:54.248037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:54.376754 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:54.376720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:56.864670 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:56.864638    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:56.976295 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:56.976265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:58.499675 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:58.499640    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:03:58.601037 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:03:58.600985    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:01.012437 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:01.012387    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:01.113234 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:01.113202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:05.657352 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:05.657300    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:05.759904 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:05.759870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:05.859480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:05.859446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:14.494685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:14.494646    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:14.598799 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:14.598767    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:17.532940 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:17.532908    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:17.635999 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:17.635961    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:20.111247 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:20.111209    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:20.212261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:20.212226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:23.771893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:23.771854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:23.901910 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:23.901878    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:24.018699 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.018664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:24.581442 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.581406    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hrqf8"]
Sep 09 20:04:24.600616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.600586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hrqf8"]
Sep 09 20:04:24.600766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.600678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8"
Sep 09 20:04:24.603506 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.603489    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m7855\""
Sep 09 20:04:24.633104 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.633068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:24.702892 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.702852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwc5z\" (UniqueName: \"kubernetes.io/projected/eb9a817b-e571-4a9f-9fc0-5308f55d1fac-kube-api-access-rwc5z\") pod \"authorino-7bcfd8f9cd-hrqf8\" (UID: \"eb9a817b-e571-4a9f-9fc0-5308f55d1fac\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8"
Sep 09 20:04:24.721638 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.721603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:24.803840 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.803783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwc5z\" (UniqueName: \"kubernetes.io/projected/eb9a817b-e571-4a9f-9fc0-5308f55d1fac-kube-api-access-rwc5z\") pod \"authorino-7bcfd8f9cd-hrqf8\" (UID: \"eb9a817b-e571-4a9f-9fc0-5308f55d1fac\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8"
Sep 09 20:04:24.807245 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.807215    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-zwjpl"]
Sep 09 20:04:24.810437 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.810420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zwjpl"
Sep 09 20:04:24.820796 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.820766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwc5z\" (UniqueName: \"kubernetes.io/projected/eb9a817b-e571-4a9f-9fc0-5308f55d1fac-kube-api-access-rwc5z\") pod \"authorino-7bcfd8f9cd-hrqf8\" (UID: \"eb9a817b-e571-4a9f-9fc0-5308f55d1fac\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8"
Sep 09 20:04:24.820909 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.820769    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zwjpl"]
Sep 09 20:04:24.904448 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.904407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv6kb\" (UniqueName: \"kubernetes.io/projected/41a96585-d3ca-4941-92d6-ccfa2c652ebe-kube-api-access-mv6kb\") pod \"authorino-59c974bd48-zwjpl\" (UID: \"41a96585-d3ca-4941-92d6-ccfa2c652ebe\") " pod="kuadrant-system/authorino-59c974bd48-zwjpl"
Sep 09 20:04:24.909215 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:24.909191    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8"
Sep 09 20:04:25.005706 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:25.005663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv6kb\" (UniqueName: \"kubernetes.io/projected/41a96585-d3ca-4941-92d6-ccfa2c652ebe-kube-api-access-mv6kb\") pod \"authorino-59c974bd48-zwjpl\" (UID: \"41a96585-d3ca-4941-92d6-ccfa2c652ebe\") " pod="kuadrant-system/authorino-59c974bd48-zwjpl"
Sep 09 20:04:25.014908 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:25.014875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv6kb\" (UniqueName: \"kubernetes.io/projected/41a96585-d3ca-4941-92d6-ccfa2c652ebe-kube-api-access-mv6kb\") pod \"authorino-59c974bd48-zwjpl\" (UID: \"41a96585-d3ca-4941-92d6-ccfa2c652ebe\") " pod="kuadrant-system/authorino-59c974bd48-zwjpl"
Sep 09 20:04:25.118724 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:25.118684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zwjpl"
Sep 09 20:04:25.178789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:25.178751    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hrqf8"]
Sep 09 20:04:25.181105 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:04:25.181072    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb9a817b_e571_4a9f_9fc0_5308f55d1fac.slice/crio-befaea891c5b585919aa32228abc7b9f383ea72e26fd96119d9672914f079370 WatchSource:0}: Error finding container befaea891c5b585919aa32228abc7b9f383ea72e26fd96119d9672914f079370: Status 404 returned error can't find the container with id befaea891c5b585919aa32228abc7b9f383ea72e26fd96119d9672914f079370
Sep 09 20:04:25.385834 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:25.385723    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zwjpl"]
Sep 09 20:04:25.388316 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:04:25.388284    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41a96585_d3ca_4941_92d6_ccfa2c652ebe.slice/crio-0611848f8390f128703a99c55ac47963368450d3fbf4c97379c553b5dea83f5b WatchSource:0}: Error finding container 0611848f8390f128703a99c55ac47963368450d3fbf4c97379c553b5dea83f5b: Status 404 returned error can't find the container with id 0611848f8390f128703a99c55ac47963368450d3fbf4c97379c553b5dea83f5b
Sep 09 20:04:25.495953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:25.495862    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:25.596916 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:25.596885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:25.910263 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:25.910229    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:25.985047 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:25.984999    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:26.145928 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:26.145888    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zwjpl" event={"ID":"41a96585-d3ca-4941-92d6-ccfa2c652ebe","Type":"ContainerStarted","Data":"0611848f8390f128703a99c55ac47963368450d3fbf4c97379c553b5dea83f5b"}
Sep 09 20:04:26.147410 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:26.147378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8" event={"ID":"eb9a817b-e571-4a9f-9fc0-5308f55d1fac","Type":"ContainerStarted","Data":"befaea891c5b585919aa32228abc7b9f383ea72e26fd96119d9672914f079370"}
Sep 09 20:04:26.321480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:26.321394    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:26.423347 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:26.423296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:27.153025 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:27.152992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zwjpl" event={"ID":"41a96585-d3ca-4941-92d6-ccfa2c652ebe","Type":"ContainerStarted","Data":"659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208"}
Sep 09 20:04:27.154639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:27.154616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8" event={"ID":"eb9a817b-e571-4a9f-9fc0-5308f55d1fac","Type":"ContainerStarted","Data":"9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad"}
Sep 09 20:04:27.173089 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:27.173026    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-zwjpl" podStartSLOduration=1.489990359 podStartE2EDuration="3.173005354s" totalImagesPullingTime="1.683014995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:04:27.171217131 +0000 UTC m=+575.759870788" watchObservedRunningTime="2026-09-09 20:04:27.173005354 +0000 UTC m=+575.761659015"
Sep 09 20:04:27.235538 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:27.235472    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8" podStartSLOduration=1.340807262 podStartE2EDuration="3.235448732s" totalImagesPullingTime="1.89464147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:04:27.190053557 +0000 UTC m=+575.778707216" watchObservedRunningTime="2026-09-09 20:04:27.235448732 +0000 UTC m=+575.824102397"
Sep 09 20:04:27.236603 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:27.236574    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hrqf8"]
Sep 09 20:04:29.163057 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:29.163014    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8" podUID="eb9a817b-e571-4a9f-9fc0-5308f55d1fac" containerName="authorino" containerID="cri-o://9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad" gracePeriod=30
Sep 09 20:04:29.415153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:29.415093    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8"
Sep 09 20:04:29.546132 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:29.546097    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwc5z\" (UniqueName: \"kubernetes.io/projected/eb9a817b-e571-4a9f-9fc0-5308f55d1fac-kube-api-access-rwc5z\") pod \"eb9a817b-e571-4a9f-9fc0-5308f55d1fac\" (UID: \"eb9a817b-e571-4a9f-9fc0-5308f55d1fac\") "
Sep 09 20:04:29.548265 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:29.548234    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb9a817b-e571-4a9f-9fc0-5308f55d1fac-kube-api-access-rwc5z" (OuterVolumeSpecName: "kube-api-access-rwc5z") pod "eb9a817b-e571-4a9f-9fc0-5308f55d1fac" (UID: "eb9a817b-e571-4a9f-9fc0-5308f55d1fac"). InnerVolumeSpecName "kube-api-access-rwc5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:04:29.647135 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:29.647093    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwc5z\" (UniqueName: \"kubernetes.io/projected/eb9a817b-e571-4a9f-9fc0-5308f55d1fac-kube-api-access-rwc5z\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:04:30.168532 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:30.168439    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb9a817b-e571-4a9f-9fc0-5308f55d1fac" containerID="9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad" exitCode=0
Sep 09 20:04:30.168532 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:30.168479    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8"
Sep 09 20:04:30.168953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:30.168535    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8" event={"ID":"eb9a817b-e571-4a9f-9fc0-5308f55d1fac","Type":"ContainerDied","Data":"9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad"}
Sep 09 20:04:30.168953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:30.168574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hrqf8" event={"ID":"eb9a817b-e571-4a9f-9fc0-5308f55d1fac","Type":"ContainerDied","Data":"befaea891c5b585919aa32228abc7b9f383ea72e26fd96119d9672914f079370"}
Sep 09 20:04:30.168953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:30.168590    2578 scope.go:117] "RemoveContainer" containerID="9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad"
Sep 09 20:04:30.176675 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:30.176655    2578 scope.go:117] "RemoveContainer" containerID="9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad"
Sep 09 20:04:30.176950 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:04:30.176933    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad\": container with ID starting with 9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad not found: ID does not exist" containerID="9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad"
Sep 09 20:04:30.177014 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:30.176961    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad"} err="failed to get container status \"9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad\": rpc error: code = NotFound desc = could not find container \"9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad\": container with ID starting with 9cf0b6e2741782a00c9fb6d44152dd6ea87cdefcf351cb32bcb5da243a5e05ad not found: ID does not exist"
Sep 09 20:04:30.186801 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:30.186770    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hrqf8"]
Sep 09 20:04:30.194742 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:30.194720    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hrqf8"]
Sep 09 20:04:31.895612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:31.895570    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb9a817b-e571-4a9f-9fc0-5308f55d1fac" path="/var/lib/kubelet/pods/eb9a817b-e571-4a9f-9fc0-5308f55d1fac/volumes"
Sep 09 20:04:33.164385 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:33.164355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:34.327220 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:34.327181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:34.427960 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:34.427919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:34.535192 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:34.535157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:37.319948 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:37.319915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:37.430113 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:37.430080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:38.673408 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:38.673370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:43.857380 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:43.857327    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:43.995851 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:43.995816    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:47.013531 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.013496    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-c6hdh"]
Sep 09 20:04:47.013964 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.013839    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb9a817b-e571-4a9f-9fc0-5308f55d1fac" containerName="authorino"
Sep 09 20:04:47.013964 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.013851    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb9a817b-e571-4a9f-9fc0-5308f55d1fac" containerName="authorino"
Sep 09 20:04:47.013964 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.013910    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb9a817b-e571-4a9f-9fc0-5308f55d1fac" containerName="authorino"
Sep 09 20:04:47.016548 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.016531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-c6hdh"
Sep 09 20:04:47.019149 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.019127    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 20:04:47.025838 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.025813    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-c6hdh"]
Sep 09 20:04:47.085533 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.085491    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff9nn\" (UniqueName: \"kubernetes.io/projected/816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4-kube-api-access-ff9nn\") pod \"authorino-6d9898bd99-c6hdh\" (UID: \"816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4\") " pod="kuadrant-system/authorino-6d9898bd99-c6hdh"
Sep 09 20:04:47.085533 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.085536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4-tls-cert\") pod \"authorino-6d9898bd99-c6hdh\" (UID: \"816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4\") " pod="kuadrant-system/authorino-6d9898bd99-c6hdh"
Sep 09 20:04:47.186597 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.186550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff9nn\" (UniqueName: \"kubernetes.io/projected/816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4-kube-api-access-ff9nn\") pod \"authorino-6d9898bd99-c6hdh\" (UID: \"816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4\") " pod="kuadrant-system/authorino-6d9898bd99-c6hdh"
Sep 09 20:04:47.186754 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.186610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4-tls-cert\") pod \"authorino-6d9898bd99-c6hdh\" (UID: \"816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4\") " pod="kuadrant-system/authorino-6d9898bd99-c6hdh"
Sep 09 20:04:47.189053 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.189022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4-tls-cert\") pod \"authorino-6d9898bd99-c6hdh\" (UID: \"816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4\") " pod="kuadrant-system/authorino-6d9898bd99-c6hdh"
Sep 09 20:04:47.196599 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.196577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff9nn\" (UniqueName: \"kubernetes.io/projected/816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4-kube-api-access-ff9nn\") pod \"authorino-6d9898bd99-c6hdh\" (UID: \"816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4\") " pod="kuadrant-system/authorino-6d9898bd99-c6hdh"
Sep 09 20:04:47.325151 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.325066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-c6hdh"
Sep 09 20:04:47.597999 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:47.597972    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-c6hdh"]
Sep 09 20:04:47.600595 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:04:47.600561    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod816edc7c_ffa4_4cd5_af7d_4ed59ffa30e4.slice/crio-da599abe928104250093a6f06e5c091d0aa30ef558890dd4918060a7bf82c0de WatchSource:0}: Error finding container da599abe928104250093a6f06e5c091d0aa30ef558890dd4918060a7bf82c0de: Status 404 returned error can't find the container with id da599abe928104250093a6f06e5c091d0aa30ef558890dd4918060a7bf82c0de
Sep 09 20:04:48.242236 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:48.242201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-c6hdh" event={"ID":"816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4","Type":"ContainerStarted","Data":"fecdb7c9c498dc482c6b5f374dd97377825d6ed5ea7100a5abb03eed4d043af7"}
Sep 09 20:04:48.242236 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:48.242240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-c6hdh" event={"ID":"816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4","Type":"ContainerStarted","Data":"da599abe928104250093a6f06e5c091d0aa30ef558890dd4918060a7bf82c0de"}
Sep 09 20:04:48.262840 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:48.262786    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-c6hdh" podStartSLOduration=1.816306177 podStartE2EDuration="2.262771471s" totalImagesPullingTime="446.465294ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:04:48.2622914 +0000 UTC m=+596.850945079" watchObservedRunningTime="2026-09-09 20:04:48.262771471 +0000 UTC m=+596.851425128"
Sep 09 20:04:48.301491 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:48.301405    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zwjpl"]
Sep 09 20:04:48.301653 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:48.301573    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-zwjpl" podUID="41a96585-d3ca-4941-92d6-ccfa2c652ebe" containerName="authorino" containerID="cri-o://659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208" gracePeriod=30
Sep 09 20:04:48.537868 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:48.537842    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zwjpl"
Sep 09 20:04:48.597539 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:48.597509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv6kb\" (UniqueName: \"kubernetes.io/projected/41a96585-d3ca-4941-92d6-ccfa2c652ebe-kube-api-access-mv6kb\") pod \"41a96585-d3ca-4941-92d6-ccfa2c652ebe\" (UID: \"41a96585-d3ca-4941-92d6-ccfa2c652ebe\") "
Sep 09 20:04:48.599606 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:48.599579    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a96585-d3ca-4941-92d6-ccfa2c652ebe-kube-api-access-mv6kb" (OuterVolumeSpecName: "kube-api-access-mv6kb") pod "41a96585-d3ca-4941-92d6-ccfa2c652ebe" (UID: "41a96585-d3ca-4941-92d6-ccfa2c652ebe"). InnerVolumeSpecName "kube-api-access-mv6kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:04:48.698084 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:48.698037    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv6kb\" (UniqueName: \"kubernetes.io/projected/41a96585-d3ca-4941-92d6-ccfa2c652ebe-kube-api-access-mv6kb\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:04:49.247857 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.247821    2578 generic.go:356] "Generic (PLEG): container finished" podID="41a96585-d3ca-4941-92d6-ccfa2c652ebe" containerID="659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208" exitCode=0
Sep 09 20:04:49.248357 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.247862    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zwjpl"
Sep 09 20:04:49.248357 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.247911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zwjpl" event={"ID":"41a96585-d3ca-4941-92d6-ccfa2c652ebe","Type":"ContainerDied","Data":"659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208"}
Sep 09 20:04:49.248357 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.247958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zwjpl" event={"ID":"41a96585-d3ca-4941-92d6-ccfa2c652ebe","Type":"ContainerDied","Data":"0611848f8390f128703a99c55ac47963368450d3fbf4c97379c553b5dea83f5b"}
Sep 09 20:04:49.248357 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.247984    2578 scope.go:117] "RemoveContainer" containerID="659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208"
Sep 09 20:04:49.256663 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.256646    2578 scope.go:117] "RemoveContainer" containerID="659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208"
Sep 09 20:04:49.256907 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:04:49.256889    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208\": container with ID starting with 659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208 not found: ID does not exist" containerID="659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208"
Sep 09 20:04:49.256971 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.256921    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208"} err="failed to get container status \"659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208\": rpc error: code = NotFound desc = could not find container \"659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208\": container with ID starting with 659fb70f7207596460523aa51ef1d4fdc58f42b8485762dd956c1524506f8208 not found: ID does not exist"
Sep 09 20:04:49.271605 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.271574    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zwjpl"]
Sep 09 20:04:49.275861 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.275841    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zwjpl"]
Sep 09 20:04:49.896455 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:49.896420    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41a96585-d3ca-4941-92d6-ccfa2c652ebe" path="/var/lib/kubelet/pods/41a96585-d3ca-4941-92d6-ccfa2c652ebe/volumes"
Sep 09 20:04:51.837548 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:51.837514    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:04:51.841350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:51.841323    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:04:52.512205 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:52.512174    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:04:52.613858 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:04:52.613813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:22.243219 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.243180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:22.361515 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.361483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:22.396901 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.396869    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-98mwf"]
Sep 09 20:05:22.397342 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.397297    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a96585-d3ca-4941-92d6-ccfa2c652ebe" containerName="authorino"
Sep 09 20:05:22.397342 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.397336    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a96585-d3ca-4941-92d6-ccfa2c652ebe" containerName="authorino"
Sep 09 20:05:22.397458 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.397447    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a96585-d3ca-4941-92d6-ccfa2c652ebe" containerName="authorino"
Sep 09 20:05:22.399543 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.399520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:22.403347 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.403301    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:05:22.404255 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.404237    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:05:22.404376 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.404272    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:05:22.404376 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.404279    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hg5lf\""
Sep 09 20:05:22.411121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.411098    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-98mwf"]
Sep 09 20:05:22.491804 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.491769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:22.571646 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.571560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5w7w\" (UniqueName: \"kubernetes.io/projected/bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b-kube-api-access-h5w7w\") pod \"seaweedfs-64568bcd49-98mwf\" (UID: \"bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b\") " pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:22.571646 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.571609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b-data\") pod \"seaweedfs-64568bcd49-98mwf\" (UID: \"bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b\") " pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:22.672959 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.672920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5w7w\" (UniqueName: \"kubernetes.io/projected/bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b-kube-api-access-h5w7w\") pod \"seaweedfs-64568bcd49-98mwf\" (UID: \"bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b\") " pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:22.673111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.672969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b-data\") pod \"seaweedfs-64568bcd49-98mwf\" (UID: \"bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b\") " pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:22.673415 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.673395    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b-data\") pod \"seaweedfs-64568bcd49-98mwf\" (UID: \"bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b\") " pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:22.685124 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.685098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5w7w\" (UniqueName: \"kubernetes.io/projected/bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b-kube-api-access-h5w7w\") pod \"seaweedfs-64568bcd49-98mwf\" (UID: \"bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b\") " pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:22.708230 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.708197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:22.808974 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.808937    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:22.933383 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:22.933334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:23.001679 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:23.001647    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-98mwf"]
Sep 09 20:05:23.005029 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:05:23.005003    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2f825a_3ff3_4a47_9b7e_1ffa22eec41b.slice/crio-ca2833496a0b2bf048b7d88f07025d3ee48ca5dfc0967d2f30ada15d73f79b95 WatchSource:0}: Error finding container ca2833496a0b2bf048b7d88f07025d3ee48ca5dfc0967d2f30ada15d73f79b95: Status 404 returned error can't find the container with id ca2833496a0b2bf048b7d88f07025d3ee48ca5dfc0967d2f30ada15d73f79b95
Sep 09 20:05:23.042472 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:23.042444    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:23.143009 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:23.142969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:23.217301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:23.217223    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:23.397033 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:23.396994    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-98mwf" event={"ID":"bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b","Type":"ContainerStarted","Data":"ca2833496a0b2bf048b7d88f07025d3ee48ca5dfc0967d2f30ada15d73f79b95"}
Sep 09 20:05:27.417831 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:27.417794    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-98mwf" event={"ID":"bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b","Type":"ContainerStarted","Data":"635bf3e0b2832fa9bc86723d884d30ada4a9db4403ad97bef5a77ce6e30fc4b8"}
Sep 09 20:05:27.418349 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:27.417861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:33.424519 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:33.424481    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-98mwf"
Sep 09 20:05:33.443028 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:33.442971    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-98mwf" podStartSLOduration=7.192046148 podStartE2EDuration="11.442953576s" totalImagesPullingTime="4.250907428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:05:27.44087939 +0000 UTC m=+636.029533047" watchObservedRunningTime="2026-09-09 20:05:33.442953576 +0000 UTC m=+642.031607234"
Sep 09 20:05:36.422283 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:36.422248    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:05:48.041052 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:05:48.041016    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:10.340969 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:10.340929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:37.772015 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:37.771983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:37.870871 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:37.870838    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:42.196827 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.196788    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:42.248865 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.248831    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-z4tn9"]
Sep 09 20:06:42.252065 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.252042    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:42.254735 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.254712    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 20:06:42.254849 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.254832    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-r8kbj\""
Sep 09 20:06:42.261977 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.261950    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-v98qv"]
Sep 09 20:06:42.265818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.265797    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-z4tn9"]
Sep 09 20:06:42.265933 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.265916    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.268671 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.268650    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:06:42.269142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.269122    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-r5lwj\""
Sep 09 20:06:42.269372 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.269354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 20:06:42.283818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.283790    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-v98qv"]
Sep 09 20:06:42.296468 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.296436    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:42.349195 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.349153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lhn\" (UniqueName: \"kubernetes.io/projected/5bce6bde-3815-41cf-916c-69f883449e09-kube-api-access-z5lhn\") pod \"model-serving-api-7b954ffc95-z4tn9\" (UID: \"5bce6bde-3815-41cf-916c-69f883449e09\") " pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:42.349408 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.349203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bce6bde-3815-41cf-916c-69f883449e09-tls-certs\") pod \"model-serving-api-7b954ffc95-z4tn9\" (UID: \"5bce6bde-3815-41cf-916c-69f883449e09\") " pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:42.349408 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.349232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d43d0471-02f4-4bcb-ba24-9f055ac3636b-cert\") pod \"odh-model-controller-7b966d7984-v98qv\" (UID: \"d43d0471-02f4-4bcb-ba24-9f055ac3636b\") " pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.349408 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.349322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43d0471-02f4-4bcb-ba24-9f055ac3636b-metrics-certs\") pod \"odh-model-controller-7b966d7984-v98qv\" (UID: \"d43d0471-02f4-4bcb-ba24-9f055ac3636b\") " pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.349408 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.349378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl22v\" (UniqueName: \"kubernetes.io/projected/d43d0471-02f4-4bcb-ba24-9f055ac3636b-kube-api-access-bl22v\") pod \"odh-model-controller-7b966d7984-v98qv\" (UID: \"d43d0471-02f4-4bcb-ba24-9f055ac3636b\") " pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.435065 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.435025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:42.450255 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.450149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lhn\" (UniqueName: \"kubernetes.io/projected/5bce6bde-3815-41cf-916c-69f883449e09-kube-api-access-z5lhn\") pod \"model-serving-api-7b954ffc95-z4tn9\" (UID: \"5bce6bde-3815-41cf-916c-69f883449e09\") " pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:42.450255 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.450198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bce6bde-3815-41cf-916c-69f883449e09-tls-certs\") pod \"model-serving-api-7b954ffc95-z4tn9\" (UID: \"5bce6bde-3815-41cf-916c-69f883449e09\") " pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:42.450255 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.450219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d43d0471-02f4-4bcb-ba24-9f055ac3636b-cert\") pod \"odh-model-controller-7b966d7984-v98qv\" (UID: \"d43d0471-02f4-4bcb-ba24-9f055ac3636b\") " pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.450255 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.450237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43d0471-02f4-4bcb-ba24-9f055ac3636b-metrics-certs\") pod \"odh-model-controller-7b966d7984-v98qv\" (UID: \"d43d0471-02f4-4bcb-ba24-9f055ac3636b\") " pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.450608 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.450271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl22v\" (UniqueName: \"kubernetes.io/projected/d43d0471-02f4-4bcb-ba24-9f055ac3636b-kube-api-access-bl22v\") pod \"odh-model-controller-7b966d7984-v98qv\" (UID: \"d43d0471-02f4-4bcb-ba24-9f055ac3636b\") " pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.450608 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:06:42.450355    2578 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 20:06:42.450608 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:06:42.450439    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bce6bde-3815-41cf-916c-69f883449e09-tls-certs podName:5bce6bde-3815-41cf-916c-69f883449e09 nodeName:}" failed. No retries permitted until 2026-09-09 20:06:42.950415509 +0000 UTC m=+711.539069149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5bce6bde-3815-41cf-916c-69f883449e09-tls-certs") pod "model-serving-api-7b954ffc95-z4tn9" (UID: "5bce6bde-3815-41cf-916c-69f883449e09") : secret "model-serving-api-tls" not found
Sep 09 20:06:42.452716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.452686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d43d0471-02f4-4bcb-ba24-9f055ac3636b-cert\") pod \"odh-model-controller-7b966d7984-v98qv\" (UID: \"d43d0471-02f4-4bcb-ba24-9f055ac3636b\") " pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.452941 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.452919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d43d0471-02f4-4bcb-ba24-9f055ac3636b-metrics-certs\") pod \"odh-model-controller-7b966d7984-v98qv\" (UID: \"d43d0471-02f4-4bcb-ba24-9f055ac3636b\") " pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.461547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.461528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl22v\" (UniqueName: \"kubernetes.io/projected/d43d0471-02f4-4bcb-ba24-9f055ac3636b-kube-api-access-bl22v\") pod \"odh-model-controller-7b966d7984-v98qv\" (UID: \"d43d0471-02f4-4bcb-ba24-9f055ac3636b\") " pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.461868 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.461849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lhn\" (UniqueName: \"kubernetes.io/projected/5bce6bde-3815-41cf-916c-69f883449e09-kube-api-access-z5lhn\") pod \"model-serving-api-7b954ffc95-z4tn9\" (UID: \"5bce6bde-3815-41cf-916c-69f883449e09\") " pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:42.537977 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.537935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:42.577889 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.577853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:42.640111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.640021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:42.851372 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.851341    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-v98qv"]
Sep 09 20:06:42.854095 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:06:42.854067    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43d0471_02f4_4bcb_ba24_9f055ac3636b.slice/crio-69c701b43dd02a2a996ae676d95675d67696bdf5a8e17be5b2985748be4baaf2 WatchSource:0}: Error finding container 69c701b43dd02a2a996ae676d95675d67696bdf5a8e17be5b2985748be4baaf2: Status 404 returned error can't find the container with id 69c701b43dd02a2a996ae676d95675d67696bdf5a8e17be5b2985748be4baaf2
Sep 09 20:06:42.855458 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.855435    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:06:42.955636 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.955600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bce6bde-3815-41cf-916c-69f883449e09-tls-certs\") pod \"model-serving-api-7b954ffc95-z4tn9\" (UID: \"5bce6bde-3815-41cf-916c-69f883449e09\") " pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:42.958006 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:42.957985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bce6bde-3815-41cf-916c-69f883449e09-tls-certs\") pod \"model-serving-api-7b954ffc95-z4tn9\" (UID: \"5bce6bde-3815-41cf-916c-69f883449e09\") " pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:43.163019 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:43.162933    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:43.441398 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:43.441369    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-z4tn9"]
Sep 09 20:06:43.444344 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:06:43.444296    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bce6bde_3815_41cf_916c_69f883449e09.slice/crio-71d2a7f5351ebd78f62a49bcf24298145c4c3bbebdf40a4d12f7f0c2aa44883b WatchSource:0}: Error finding container 71d2a7f5351ebd78f62a49bcf24298145c4c3bbebdf40a4d12f7f0c2aa44883b: Status 404 returned error can't find the container with id 71d2a7f5351ebd78f62a49bcf24298145c4c3bbebdf40a4d12f7f0c2aa44883b
Sep 09 20:06:43.477856 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:43.477824    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:43.567568 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:43.567536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:06:43.734995 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:43.734893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-v98qv" event={"ID":"d43d0471-02f4-4bcb-ba24-9f055ac3636b","Type":"ContainerStarted","Data":"69c701b43dd02a2a996ae676d95675d67696bdf5a8e17be5b2985748be4baaf2"}
Sep 09 20:06:43.736238 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:43.736205    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-z4tn9" event={"ID":"5bce6bde-3815-41cf-916c-69f883449e09","Type":"ContainerStarted","Data":"71d2a7f5351ebd78f62a49bcf24298145c4c3bbebdf40a4d12f7f0c2aa44883b"}
Sep 09 20:06:46.752837 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:46.752797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-v98qv" event={"ID":"d43d0471-02f4-4bcb-ba24-9f055ac3636b","Type":"ContainerStarted","Data":"3c814cfa3cff77676877f3fa8b58d6d24f6c654c435ee55c8f926f9f0b344c96"}
Sep 09 20:06:46.753437 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:46.752862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:46.754237 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:46.754213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-z4tn9" event={"ID":"5bce6bde-3815-41cf-916c-69f883449e09","Type":"ContainerStarted","Data":"212228f25c0a07b9e66c60520c5e538fbc977005ccbef9e9daba6249ba04fee1"}
Sep 09 20:06:46.754331 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:46.754270    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:06:46.773668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:46.773614    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-v98qv" podStartSLOduration=1.585257746 podStartE2EDuration="4.773597233s" totalImagesPullingTime="3.188339487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:06:46.771962147 +0000 UTC m=+715.360615804" watchObservedRunningTime="2026-09-09 20:06:46.773597233 +0000 UTC m=+715.362250886"
Sep 09 20:06:46.790570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:46.790513    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-z4tn9" podStartSLOduration=2.202741215 podStartE2EDuration="4.790496038s" totalImagesPullingTime="2.587754823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:06:46.789411356 +0000 UTC m=+715.378065012" watchObservedRunningTime="2026-09-09 20:06:46.790496038 +0000 UTC m=+715.379149695"
Sep 09 20:06:57.761642 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:57.761611    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-v98qv"
Sep 09 20:06:57.763403 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:06:57.763385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-z4tn9"
Sep 09 20:07:24.101593 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:07:24.101553    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:07:24.203003 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:07:24.202973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:07:50.455803 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:07:50.455769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:07:50.537062 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:07:50.537029    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:18.379200 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:18.379109    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:18.576759 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:18.576721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:31.563672 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:31.563623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:31.664298 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:31.664258    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:40.037834 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.037795    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"]
Sep 09 20:08:40.041316 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.041287    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.045283 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.044778    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 09 20:08:40.046748 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.046045    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 09 20:08:40.046748 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.046065    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-4m7n8\""
Sep 09 20:08:40.047219 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.046966    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 09 20:08:40.058340 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.058293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"]
Sep 09 20:08:40.079146 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.079116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:40.152543 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.152505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-dshm\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.152711 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.152569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-home\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.152711 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.152649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb801a4-7256-41ab-9281-1a49a1739298-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.152711 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.152686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5hg\" (UniqueName: \"kubernetes.io/projected/0bb801a4-7256-41ab-9281-1a49a1739298-kube-api-access-2n5hg\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.152847 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.152731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.152847 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.152797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.253860 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.253829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.254055 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.253877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-dshm\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.254055 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.253916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-home\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.254055 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.253932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb801a4-7256-41ab-9281-1a49a1739298-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.254055 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.253948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5hg\" (UniqueName: \"kubernetes.io/projected/0bb801a4-7256-41ab-9281-1a49a1739298-kube-api-access-2n5hg\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.254055 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.253972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.254402 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.254378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.254471 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.254399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-home\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.254592 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.254564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.256303 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.256278    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-dshm\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.256607 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.256584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb801a4-7256-41ab-9281-1a49a1739298-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.267919 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.267897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5hg\" (UniqueName: \"kubernetes.io/projected/0bb801a4-7256-41ab-9281-1a49a1739298-kube-api-access-2n5hg\") pod \"fc-smoke-test-kserve-7f4687897-49hln\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.359537 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.359499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:40.454552 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.454507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:08:40.631721 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:40.631695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"]
Sep 09 20:08:40.633781 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:08:40.633745    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb801a4_7256_41ab_9281_1a49a1739298.slice/crio-5d4092782ccf828560ef6faab6851d096e085016456c6b5a9a8bfee464288c16 WatchSource:0}: Error finding container 5d4092782ccf828560ef6faab6851d096e085016456c6b5a9a8bfee464288c16: Status 404 returned error can't find the container with id 5d4092782ccf828560ef6faab6851d096e085016456c6b5a9a8bfee464288c16
Sep 09 20:08:41.252599 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:41.252518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln" event={"ID":"0bb801a4-7256-41ab-9281-1a49a1739298","Type":"ContainerStarted","Data":"5d4092782ccf828560ef6faab6851d096e085016456c6b5a9a8bfee464288c16"}
Sep 09 20:08:43.278976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:43.278938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln" event={"ID":"0bb801a4-7256-41ab-9281-1a49a1739298","Type":"ContainerStarted","Data":"67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e"}
Sep 09 20:08:43.299897 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:43.299839    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln" podStartSLOduration=1.350823017 podStartE2EDuration="3.299822633s" totalImagesPullingTime="1.948999616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:08:43.298198464 +0000 UTC m=+831.886852120" watchObservedRunningTime="2026-09-09 20:08:43.299822633 +0000 UTC m=+831.888476290"
Sep 09 20:08:43.360504 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:43.360459    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:50.360598 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:50.360560    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:50.373385 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:50.373349    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:50.807420 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:50.807289    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 09 20:08:50.807576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:50.807406    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" podUID="870011d5-3b7b-4211-8007-9f917ffa1944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 09 20:08:51.316808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:51.316771    2578 generic.go:356] "Generic (PLEG): container finished" podID="870011d5-3b7b-4211-8007-9f917ffa1944" containerID="0d3912d8e4b99260b9702d66303a303fdddc93c4ca4166e8dc67f6454691bd07" exitCode=0
Sep 09 20:08:51.317013 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:51.316845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" event={"ID":"870011d5-3b7b-4211-8007-9f917ffa1944","Type":"ContainerDied","Data":"0d3912d8e4b99260b9702d66303a303fdddc93c4ca4166e8dc67f6454691bd07"}
Sep 09 20:08:51.317403 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:51.317385    2578 scope.go:117] "RemoveContainer" containerID="0d3912d8e4b99260b9702d66303a303fdddc93c4ca4166e8dc67f6454691bd07"
Sep 09 20:08:51.318358 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:51.318302    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:08:51.318430 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:51.318417    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:08:51.318477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:51.318462    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:08:51.328876 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:51.328848    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:08:52.324091 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:52.324046    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" event={"ID":"870011d5-3b7b-4211-8007-9f917ffa1944","Type":"ContainerStarted","Data":"7e11a6fd0eed92d0983206b3024db85a1253a9ef3dcc588c549eb12dafdaa035"}
Sep 09 20:08:52.700951 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:52.700903    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:08:52.701204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:52.701179    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 09 20:08:52.701260 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:52.701236    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" podUID="870011d5-3b7b-4211-8007-9f917ffa1944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 09 20:08:53.701650 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:53.701610    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 09 20:08:53.702014 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:53.701677    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" podUID="870011d5-3b7b-4211-8007-9f917ffa1944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 09 20:08:54.701865 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:54.701819    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 09 20:08:54.702365 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:54.701880    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" podUID="870011d5-3b7b-4211-8007-9f917ffa1944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 09 20:08:55.701533 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:55.701490    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 09 20:08:55.701757 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:55.701560    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" podUID="870011d5-3b7b-4211-8007-9f917ffa1944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 09 20:08:56.701636 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:56.701592    2578 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 09 20:08:56.701993 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:56.701658    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv" podUID="870011d5-3b7b-4211-8007-9f917ffa1944" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 09 20:08:57.706703 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:57.706666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:08:57.707146 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:57.707076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:08:57.708022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:57.708002    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv"
Sep 09 20:08:59.693595 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:08:59.693546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:09:03.542354 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:03.542288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:09:18.298092 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:18.298048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:09:18.398759 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:18.398713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:09:22.455154 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.455120    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"]
Sep 09 20:09:22.464114 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:09:22.464072    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:09:22.464114 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:09:22.464114 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:09:22.464114 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e" cmd=["/bin/sleep","15"]
Sep 09 20:09:22.464463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.464126    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln" message="" err=<
Sep 09 20:09:22.464463 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:09:22.464463 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:09:22.464463 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:09:22.464463 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:09:22.464139    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:09:22.464463 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:09:22.464463 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:09:22.464463 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln" podUID="0bb801a4-7256-41ab-9281-1a49a1739298" containerName="main" containerID="cri-o://67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e"
Sep 09 20:09:22.464463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.464173    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln" podUID="0bb801a4-7256-41ab-9281-1a49a1739298" containerName="main" containerID="cri-o://67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e" gracePeriod=60
Sep 09 20:09:22.467991 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.467956    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:09:22.568596 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.568547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:09:22.809454 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.809428    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:09:22.920698 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.920654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-dshm\") pod \"0bb801a4-7256-41ab-9281-1a49a1739298\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") "
Sep 09 20:09:22.920880 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.920707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-model-cache\") pod \"0bb801a4-7256-41ab-9281-1a49a1739298\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") "
Sep 09 20:09:22.920880 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.920746    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n5hg\" (UniqueName: \"kubernetes.io/projected/0bb801a4-7256-41ab-9281-1a49a1739298-kube-api-access-2n5hg\") pod \"0bb801a4-7256-41ab-9281-1a49a1739298\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") "
Sep 09 20:09:22.920880 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.920792    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-tmp-dir\") pod \"0bb801a4-7256-41ab-9281-1a49a1739298\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") "
Sep 09 20:09:22.920880 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.920824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb801a4-7256-41ab-9281-1a49a1739298-tls-certs\") pod \"0bb801a4-7256-41ab-9281-1a49a1739298\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") "
Sep 09 20:09:22.921089 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.921050    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-home\") pod \"0bb801a4-7256-41ab-9281-1a49a1739298\" (UID: \"0bb801a4-7256-41ab-9281-1a49a1739298\") "
Sep 09 20:09:22.921089 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.921051    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bb801a4-7256-41ab-9281-1a49a1739298" (UID: "0bb801a4-7256-41ab-9281-1a49a1739298"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:09:22.921281 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.921251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bb801a4-7256-41ab-9281-1a49a1739298" (UID: "0bb801a4-7256-41ab-9281-1a49a1739298"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:09:22.921600 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.921374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-home" (OuterVolumeSpecName: "home") pod "0bb801a4-7256-41ab-9281-1a49a1739298" (UID: "0bb801a4-7256-41ab-9281-1a49a1739298"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:09:22.921600 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.921377    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:09:22.923101 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.923067    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-dshm" (OuterVolumeSpecName: "dshm") pod "0bb801a4-7256-41ab-9281-1a49a1739298" (UID: "0bb801a4-7256-41ab-9281-1a49a1739298"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:09:22.923262 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.923241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb801a4-7256-41ab-9281-1a49a1739298-kube-api-access-2n5hg" (OuterVolumeSpecName: "kube-api-access-2n5hg") pod "0bb801a4-7256-41ab-9281-1a49a1739298" (UID: "0bb801a4-7256-41ab-9281-1a49a1739298"). InnerVolumeSpecName "kube-api-access-2n5hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:09:22.923411 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:22.923386    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb801a4-7256-41ab-9281-1a49a1739298-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bb801a4-7256-41ab-9281-1a49a1739298" (UID: "0bb801a4-7256-41ab-9281-1a49a1739298"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:09:23.022653 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.022561    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:09:23.022653 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.022596    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:09:23.022653 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.022605    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb801a4-7256-41ab-9281-1a49a1739298-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:09:23.022653 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.022616    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n5hg\" (UniqueName: \"kubernetes.io/projected/0bb801a4-7256-41ab-9281-1a49a1739298-kube-api-access-2n5hg\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:09:23.022653 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.022625    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb801a4-7256-41ab-9281-1a49a1739298-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:09:23.464173 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.464138    2578 generic.go:356] "Generic (PLEG): container finished" podID="0bb801a4-7256-41ab-9281-1a49a1739298" containerID="67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e" exitCode=0
Sep 09 20:09:23.464583 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.464188    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"
Sep 09 20:09:23.464583 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.464216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln" event={"ID":"0bb801a4-7256-41ab-9281-1a49a1739298","Type":"ContainerDied","Data":"67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e"}
Sep 09 20:09:23.464583 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.464267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln" event={"ID":"0bb801a4-7256-41ab-9281-1a49a1739298","Type":"ContainerDied","Data":"5d4092782ccf828560ef6faab6851d096e085016456c6b5a9a8bfee464288c16"}
Sep 09 20:09:23.464583 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.464300    2578 scope.go:117] "RemoveContainer" containerID="67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e"
Sep 09 20:09:23.473242 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.473223    2578 scope.go:117] "RemoveContainer" containerID="67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e"
Sep 09 20:09:23.473583 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:09:23.473562    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e\": container with ID starting with 67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e not found: ID does not exist" containerID="67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e"
Sep 09 20:09:23.473654 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.473592    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e"} err="failed to get container status \"67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e\": rpc error: code = NotFound desc = could not find container \"67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e\": container with ID starting with 67a5dce2c183794eccefdc394f18972d6eeb01607c5be41073ef218b9e13831e not found: ID does not exist"
Sep 09 20:09:23.487228 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.487204    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"]
Sep 09 20:09:23.492090 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.492068    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-49hln"]
Sep 09 20:09:23.895741 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:23.895704    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb801a4-7256-41ab-9281-1a49a1739298" path="/var/lib/kubelet/pods/0bb801a4-7256-41ab-9281-1a49a1739298/volumes"
Sep 09 20:09:51.864841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:51.864813    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:09:51.866976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:09:51.866953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:10:12.553275 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:12.553226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:12.650378 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:12.650328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:18.911380 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:18.911346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:24.769223 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:24.769186    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:24.842137 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:24.842098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:26.437075 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.437045    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"]
Sep 09 20:10:26.437476 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.437450    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb801a4-7256-41ab-9281-1a49a1739298" containerName="main"
Sep 09 20:10:26.437476 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.437463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb801a4-7256-41ab-9281-1a49a1739298" containerName="main"
Sep 09 20:10:26.437550 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.437530    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb801a4-7256-41ab-9281-1a49a1739298" containerName="main"
Sep 09 20:10:26.439575 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.439557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.443072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.443049    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-sfpck\""
Sep 09 20:10:26.443170 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.443117    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 09 20:10:26.443231 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.443215    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 09 20:10:26.443412 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.443396    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 09 20:10:26.450919 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.450896    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"]
Sep 09 20:10:26.469434 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.469402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:26.524665 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.524625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa83328d-694c-4577-9462-e289481ccbfb-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.524665 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.524663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9pv\" (UniqueName: \"kubernetes.io/projected/fa83328d-694c-4577-9462-e289481ccbfb-kube-api-access-5m9pv\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.524896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.524731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-home\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.524896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.524827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.524896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.524847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.525002 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.524896    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.625474 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.625437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.625657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.625479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.625657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.625601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.625764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.625687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa83328d-694c-4577-9462-e289481ccbfb-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.625764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.625704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9pv\" (UniqueName: \"kubernetes.io/projected/fa83328d-694c-4577-9462-e289481ccbfb-kube-api-access-5m9pv\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.625764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.625729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-home\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.626009 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.625988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.626101 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.626059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.626146 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.626106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-home\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.627875 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.627853    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.628099 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.628079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa83328d-694c-4577-9462-e289481ccbfb-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.637657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.637628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9pv\" (UniqueName: \"kubernetes.io/projected/fa83328d-694c-4577-9462-e289481ccbfb-kube-api-access-5m9pv\") pod \"fc-concurrency-test-kserve-b5d44467c-sfls7\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:26.684776 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.684745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:26.749948 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:26.749865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:27.017846 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:27.017754    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"]
Sep 09 20:10:27.020596 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:10:27.020567    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa83328d_694c_4577_9462_e289481ccbfb.slice/crio-fc7d8d124a92b5aaf009374de73ecbd8443d8596cd51dc2ad92d71225f33ea43 WatchSource:0}: Error finding container fc7d8d124a92b5aaf009374de73ecbd8443d8596cd51dc2ad92d71225f33ea43: Status 404 returned error can't find the container with id fc7d8d124a92b5aaf009374de73ecbd8443d8596cd51dc2ad92d71225f33ea43
Sep 09 20:10:27.736507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:27.736469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7" event={"ID":"fa83328d-694c-4577-9462-e289481ccbfb","Type":"ContainerStarted","Data":"62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e"}
Sep 09 20:10:27.736989 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:27.736522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7" event={"ID":"fa83328d-694c-4577-9462-e289481ccbfb","Type":"ContainerStarted","Data":"fc7d8d124a92b5aaf009374de73ecbd8443d8596cd51dc2ad92d71225f33ea43"}
Sep 09 20:10:27.750474 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:27.750441    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:27.756729 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:27.756674    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7" podStartSLOduration=1.75662532 podStartE2EDuration="1.75662532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:27.756229794 +0000 UTC m=+936.344883448" watchObservedRunningTime="2026-09-09 20:10:27.75662532 +0000 UTC m=+936.345278979"
Sep 09 20:10:31.302588 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:31.302548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:10:36.750731 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:36.750639    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:36.763175 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:36.763139    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:10:36.787359 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:10:36.787326    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:11:05.913331 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:05.913278    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"]
Sep 09 20:11:05.926103 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:11:05.926064    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:11:05.926103 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:11:05.926103 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:11:05.926103 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e" cmd=["/bin/sleep","15"]
Sep 09 20:11:05.926529 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:05.926499    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7" message="" err=<
Sep 09 20:11:05.926529 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:11:05.926529 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:11:05.926529 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:11:05.926745 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:11:05.926531    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:11:05.926745 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:11:05.926745 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:11:05.926745 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7" podUID="fa83328d-694c-4577-9462-e289481ccbfb" containerName="main" containerID="cri-o://62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e"
Sep 09 20:11:05.926745 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:05.926592    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7" podUID="fa83328d-694c-4577-9462-e289481ccbfb" containerName="main" containerID="cri-o://62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e" gracePeriod=60
Sep 09 20:11:05.927462 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:05.927186    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:06.286867 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.286836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:11:06.363647 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.363608    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-model-cache\") pod \"fa83328d-694c-4577-9462-e289481ccbfb\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") "
Sep 09 20:11:06.363833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.363653    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-tmp-dir\") pod \"fa83328d-694c-4577-9462-e289481ccbfb\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") "
Sep 09 20:11:06.363833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.363704    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-dshm\") pod \"fa83328d-694c-4577-9462-e289481ccbfb\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") "
Sep 09 20:11:06.363833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.363779    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa83328d-694c-4577-9462-e289481ccbfb-tls-certs\") pod \"fa83328d-694c-4577-9462-e289481ccbfb\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") "
Sep 09 20:11:06.363833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.363802    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-home\") pod \"fa83328d-694c-4577-9462-e289481ccbfb\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") "
Sep 09 20:11:06.364038 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.363836    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m9pv\" (UniqueName: \"kubernetes.io/projected/fa83328d-694c-4577-9462-e289481ccbfb-kube-api-access-5m9pv\") pod \"fa83328d-694c-4577-9462-e289481ccbfb\" (UID: \"fa83328d-694c-4577-9462-e289481ccbfb\") "
Sep 09 20:11:06.364038 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.363873    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa83328d-694c-4577-9462-e289481ccbfb" (UID: "fa83328d-694c-4577-9462-e289481ccbfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:11:06.364038 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.363900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa83328d-694c-4577-9462-e289481ccbfb" (UID: "fa83328d-694c-4577-9462-e289481ccbfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:11:06.364164 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.364044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-home" (OuterVolumeSpecName: "home") pod "fa83328d-694c-4577-9462-e289481ccbfb" (UID: "fa83328d-694c-4577-9462-e289481ccbfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:11:06.364164 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.364065    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:11:06.364164 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.364084    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:11:06.366053 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.366020    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-dshm" (OuterVolumeSpecName: "dshm") pod "fa83328d-694c-4577-9462-e289481ccbfb" (UID: "fa83328d-694c-4577-9462-e289481ccbfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:11:06.366194 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.366103    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa83328d-694c-4577-9462-e289481ccbfb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa83328d-694c-4577-9462-e289481ccbfb" (UID: "fa83328d-694c-4577-9462-e289481ccbfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:11:06.366194 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.366096    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa83328d-694c-4577-9462-e289481ccbfb-kube-api-access-5m9pv" (OuterVolumeSpecName: "kube-api-access-5m9pv") pod "fa83328d-694c-4577-9462-e289481ccbfb" (UID: "fa83328d-694c-4577-9462-e289481ccbfb"). InnerVolumeSpecName "kube-api-access-5m9pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:11:06.465149 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.465061    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa83328d-694c-4577-9462-e289481ccbfb-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:11:06.465149 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.465097    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:11:06.465149 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.465107    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m9pv\" (UniqueName: \"kubernetes.io/projected/fa83328d-694c-4577-9462-e289481ccbfb-kube-api-access-5m9pv\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:11:06.465149 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.465117    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa83328d-694c-4577-9462-e289481ccbfb-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:11:06.910656 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.910608    2578 generic.go:356] "Generic (PLEG): container finished" podID="fa83328d-694c-4577-9462-e289481ccbfb" containerID="62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e" exitCode=0
Sep 09 20:11:06.910841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.910675    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"
Sep 09 20:11:06.910841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.910687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7" event={"ID":"fa83328d-694c-4577-9462-e289481ccbfb","Type":"ContainerDied","Data":"62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e"}
Sep 09 20:11:06.910841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.910745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7" event={"ID":"fa83328d-694c-4577-9462-e289481ccbfb","Type":"ContainerDied","Data":"fc7d8d124a92b5aaf009374de73ecbd8443d8596cd51dc2ad92d71225f33ea43"}
Sep 09 20:11:06.910841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.910767    2578 scope.go:117] "RemoveContainer" containerID="62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e"
Sep 09 20:11:06.920042 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.920020    2578 scope.go:117] "RemoveContainer" containerID="62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e"
Sep 09 20:11:06.920400 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:11:06.920379    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e\": container with ID starting with 62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e not found: ID does not exist" containerID="62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e"
Sep 09 20:11:06.920461 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.920408    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e"} err="failed to get container status \"62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e\": rpc error: code = NotFound desc = could not find container \"62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e\": container with ID starting with 62ad13f19a84373c585938ae955d5d5ec43c02711247c641c0d1e8801250713e not found: ID does not exist"
Sep 09 20:11:06.931962 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.931936    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"]
Sep 09 20:11:06.936974 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:06.936949    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-sfls7"]
Sep 09 20:11:07.590626 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:07.590591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:07.896560 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:07.896474    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa83328d-694c-4577-9462-e289481ccbfb" path="/var/lib/kubelet/pods/fa83328d-694c-4577-9462-e289481ccbfb/volumes"
Sep 09 20:11:44.689110 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:44.689076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:55.779379 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:55.779347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:55.880978 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:55.880947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:55.984147 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:55.984119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:56.024871 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.024839    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"]
Sep 09 20:11:56.025201 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.025183    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa83328d-694c-4577-9462-e289481ccbfb" containerName="main"
Sep 09 20:11:56.025280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.025203    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa83328d-694c-4577-9462-e289481ccbfb" containerName="main"
Sep 09 20:11:56.025359 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.025331    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa83328d-694c-4577-9462-e289481ccbfb" containerName="main"
Sep 09 20:11:56.027687 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.027665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.030276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.030219    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 09 20:11:56.030501 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.030259    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 09 20:11:56.030618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.030291    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:11:56.030618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.030335    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-zsgf2\""
Sep 09 20:11:56.040510 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.040488    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"]
Sep 09 20:11:56.077409 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.077369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.077605 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.077432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d87c34-695d-4482-8df5-0b07ab7378b3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.077605 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.077468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.077605 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.077546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.077605 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.077573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjwc\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-kube-api-access-gsjwc\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.077605 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.077604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.077870 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.077647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.077870 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.077683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.077870 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.077733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.083248 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.083218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:56.178484 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.178448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.178685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.178493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.178685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.178515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d87c34-695d-4482-8df5-0b07ab7378b3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.178685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.178537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.178685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.178607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.178685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.178634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjwc\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-kube-api-access-gsjwc\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.178966 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.178769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.178966 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.178826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.178966 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.178857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.179083 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.179015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.179302 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.179274    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.179441 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.179420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.179547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.179521    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.179848 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.179823    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:56.179949 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.179845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d87c34-695d-4482-8df5-0b07ab7378b3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.181575 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.181552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.181659 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.181642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.187397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.187369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.187948 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.187924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjwc\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-kube-api-access-gsjwc\") pod \"router-gateway-1-openshift-default-5944c46dd4-r9fbv\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.280867 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.280748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:56.339435 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.339402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:56.609893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.609864    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"]
Sep 09 20:11:56.611832 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:11:56.611808    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d87c34_695d_4482_8df5_0b07ab7378b3.slice/crio-5efdb2fe851e55b31aa82b8fce13361c43cf9173b13c0d5b1f57b9a7c691eb91 WatchSource:0}: Error finding container 5efdb2fe851e55b31aa82b8fce13361c43cf9173b13c0d5b1f57b9a7c691eb91: Status 404 returned error can't find the container with id 5efdb2fe851e55b31aa82b8fce13361c43cf9173b13c0d5b1f57b9a7c691eb91
Sep 09 20:11:56.613913 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.613879    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:11:56.614007 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.613943    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:11:56.614007 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:56.613976    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:11:57.138414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:57.138374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv" event={"ID":"88d87c34-695d-4482-8df5-0b07ab7378b3","Type":"ContainerStarted","Data":"545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd"}
Sep 09 20:11:57.138873 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:57.138445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv" event={"ID":"88d87c34-695d-4482-8df5-0b07ab7378b3","Type":"ContainerStarted","Data":"5efdb2fe851e55b31aa82b8fce13361c43cf9173b13c0d5b1f57b9a7c691eb91"}
Sep 09 20:11:57.158362 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:57.158287    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv" podStartSLOduration=2.158267448 podStartE2EDuration="2.158267448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:57.156706015 +0000 UTC m=+1025.745359672" watchObservedRunningTime="2026-09-09 20:11:57.158267448 +0000 UTC m=+1025.746921107"
Sep 09 20:11:57.340641 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:57.340603    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:57.341958 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:57.341929    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv" podUID="88d87c34-695d-4482-8df5-0b07ab7378b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 09 20:11:58.344334 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:58.344281    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:59.147451 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:59.147413    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:59.148261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:59.148245    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:11:59.739068 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:11:59.739038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:01.987302 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:01.987265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:04.754191 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:04.754143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:04.854956 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:04.854926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:06.026650 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:06.026616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:07.623390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:07.623357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:07.688087 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:07.688051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:08.261351 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.261296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:08.364936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.364900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:08.468710 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.468676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:08.538656 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.538569    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"]
Sep 09 20:12:08.541010 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.540989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.548046 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.548018    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 09 20:12:08.548497 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.548476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 09 20:12:08.548667 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.548643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-qx5bw\""
Sep 09 20:12:08.548826 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.548533    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:12:08.562097 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.562070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"]
Sep 09 20:12:08.590280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.590252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.590444 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.590287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.590444 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.590320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.590516 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.590474    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66hr\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-kube-api-access-x66hr\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.590554 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.590515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.590554 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.590545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.590634 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.590584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.590673 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.590638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.590707 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.590672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5649f61c-86b0-440a-9c52-ac868ec5ac45-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.617272 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.617235    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:08.691290 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.691290 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.691852 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.691852 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.691852 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5649f61c-86b0-440a-9c52-ac868ec5ac45-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.691852 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.691852 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.691852 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.691852 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66hr\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-kube-api-access-x66hr\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.691852 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.692413 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.692386    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.692571 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.691883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.692799 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.692768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5649f61c-86b0-440a-9c52-ac868ec5ac45-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.692970 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.692874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.694767 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.694734    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.699373 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.699339    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.701084 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.701064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.701699 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.701676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66hr\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-kube-api-access-x66hr\") pod \"router-gateway-1-openshift-default-66bbccc5fb-gsdjv\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:08.710707 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.710684    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:08.864100 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:08.863794    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:09.135009 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:09.134985    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"]
Sep 09 20:12:09.136773 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:12:09.136747    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5649f61c_86b0_440a_9c52_ac868ec5ac45.slice/crio-e4a499c37e0288d3cadbb2bb3c9a8d75bc899a64abce46fe02a7c12eb1a3421b WatchSource:0}: Error finding container e4a499c37e0288d3cadbb2bb3c9a8d75bc899a64abce46fe02a7c12eb1a3421b: Status 404 returned error can't find the container with id e4a499c37e0288d3cadbb2bb3c9a8d75bc899a64abce46fe02a7c12eb1a3421b
Sep 09 20:12:09.138965 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:09.138924    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:12:09.139061 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:09.139009    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:12:09.139061 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:09.139051    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:12:09.189750 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:09.189720    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv" event={"ID":"5649f61c-86b0-440a-9c52-ac868ec5ac45","Type":"ContainerStarted","Data":"e4a499c37e0288d3cadbb2bb3c9a8d75bc899a64abce46fe02a7c12eb1a3421b"}
Sep 09 20:12:10.195709 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:10.195677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv" event={"ID":"5649f61c-86b0-440a-9c52-ac868ec5ac45","Type":"ContainerStarted","Data":"dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a"}
Sep 09 20:12:10.218652 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:10.218602    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv" podStartSLOduration=2.218579173 podStartE2EDuration="2.218579173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:10.217718168 +0000 UTC m=+1038.806371826" watchObservedRunningTime="2026-09-09 20:12:10.218579173 +0000 UTC m=+1038.807232830"
Sep 09 20:12:10.867343 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:10.867283    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:10.871213 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:10.871181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:11.200755 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:11.200672    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:11.201680 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:11.201653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:11.540462 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:11.540432    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:12.089428 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:12.089390    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:13.232444 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:13.232403    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"]
Sep 09 20:12:13.232907 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:13.232639    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv" podUID="88d87c34-695d-4482-8df5-0b07ab7378b3" containerName="istio-proxy" containerID="cri-o://545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd" gracePeriod=30
Sep 09 20:12:13.296594 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:13.296556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:15.190503 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.190455    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv" podUID="88d87c34-695d-4482-8df5-0b07ab7378b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:12:15.676799 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.676771    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:12:15.755163 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755133    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-token\") pod \"88d87c34-695d-4482-8df5-0b07ab7378b3\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") "
Sep 09 20:12:15.755304 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755181    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-podinfo\") pod \"88d87c34-695d-4482-8df5-0b07ab7378b3\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") "
Sep 09 20:12:15.755304 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755201    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-socket\") pod \"88d87c34-695d-4482-8df5-0b07ab7378b3\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") "
Sep 09 20:12:15.755304 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755250    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsjwc\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-kube-api-access-gsjwc\") pod \"88d87c34-695d-4482-8df5-0b07ab7378b3\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") "
Sep 09 20:12:15.755304 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755270    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-credential-socket\") pod \"88d87c34-695d-4482-8df5-0b07ab7378b3\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") "
Sep 09 20:12:15.755304 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d87c34-695d-4482-8df5-0b07ab7378b3-istiod-ca-cert\") pod \"88d87c34-695d-4482-8df5-0b07ab7378b3\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") "
Sep 09 20:12:15.755618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-envoy\") pod \"88d87c34-695d-4482-8df5-0b07ab7378b3\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") "
Sep 09 20:12:15.755618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-certs\") pod \"88d87c34-695d-4482-8df5-0b07ab7378b3\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") "
Sep 09 20:12:15.755618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-data\") pod \"88d87c34-695d-4482-8df5-0b07ab7378b3\" (UID: \"88d87c34-695d-4482-8df5-0b07ab7378b3\") "
Sep 09 20:12:15.755618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755554    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88d87c34-695d-4482-8df5-0b07ab7378b3" (UID: "88d87c34-695d-4482-8df5-0b07ab7378b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:15.755875 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755696    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:15.755875 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755801    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88d87c34-695d-4482-8df5-0b07ab7378b3" (UID: "88d87c34-695d-4482-8df5-0b07ab7378b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:15.755985 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.755901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88d87c34-695d-4482-8df5-0b07ab7378b3" (UID: "88d87c34-695d-4482-8df5-0b07ab7378b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:15.756251 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.756219    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d87c34-695d-4482-8df5-0b07ab7378b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88d87c34-695d-4482-8df5-0b07ab7378b3" (UID: "88d87c34-695d-4482-8df5-0b07ab7378b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:15.756387 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.756293    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "88d87c34-695d-4482-8df5-0b07ab7378b3" (UID: "88d87c34-695d-4482-8df5-0b07ab7378b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:15.757958 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.757799    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-kube-api-access-gsjwc" (OuterVolumeSpecName: "kube-api-access-gsjwc") pod "88d87c34-695d-4482-8df5-0b07ab7378b3" (UID: "88d87c34-695d-4482-8df5-0b07ab7378b3"). InnerVolumeSpecName "kube-api-access-gsjwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:15.757958 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.757905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88d87c34-695d-4482-8df5-0b07ab7378b3" (UID: "88d87c34-695d-4482-8df5-0b07ab7378b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:12:15.757958 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.757946    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "88d87c34-695d-4482-8df5-0b07ab7378b3" (UID: "88d87c34-695d-4482-8df5-0b07ab7378b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:15.758157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.758126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88d87c34-695d-4482-8df5-0b07ab7378b3" (UID: "88d87c34-695d-4482-8df5-0b07ab7378b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:15.857097 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.857061    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsjwc\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-kube-api-access-gsjwc\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:15.857097 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.857091    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-credential-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:15.857097 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.857101    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88d87c34-695d-4482-8df5-0b07ab7378b3-istiod-ca-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:15.857336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.857111    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-envoy\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:15.857336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.857119    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-workload-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:15.857336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.857127    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-data\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:15.857336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.857133    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-token\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:15.857336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:15.857140    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88d87c34-695d-4482-8df5-0b07ab7378b3-istio-podinfo\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:16.224670 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:16.224635    2578 generic.go:356] "Generic (PLEG): container finished" podID="88d87c34-695d-4482-8df5-0b07ab7378b3" containerID="545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd" exitCode=0
Sep 09 20:12:16.225039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:16.224678    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"
Sep 09 20:12:16.225039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:16.224680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv" event={"ID":"88d87c34-695d-4482-8df5-0b07ab7378b3","Type":"ContainerDied","Data":"545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd"}
Sep 09 20:12:16.225039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:16.224811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv" event={"ID":"88d87c34-695d-4482-8df5-0b07ab7378b3","Type":"ContainerDied","Data":"5efdb2fe851e55b31aa82b8fce13361c43cf9173b13c0d5b1f57b9a7c691eb91"}
Sep 09 20:12:16.225039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:16.224845    2578 scope.go:117] "RemoveContainer" containerID="545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd"
Sep 09 20:12:16.233251 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:16.233231    2578 scope.go:117] "RemoveContainer" containerID="545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd"
Sep 09 20:12:16.233542 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:12:16.233524    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd\": container with ID starting with 545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd not found: ID does not exist" containerID="545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd"
Sep 09 20:12:16.233594 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:16.233550    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd"} err="failed to get container status \"545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd\": rpc error: code = NotFound desc = could not find container \"545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd\": container with ID starting with 545b35b2ede66e7835c1c0f4eee9f02f2b0df7140c0649d5930f4b904d1b24fd not found: ID does not exist"
Sep 09 20:12:16.242557 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:16.242534    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"]
Sep 09 20:12:16.247070 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:16.247049    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-r9fbv"]
Sep 09 20:12:17.632795 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:17.632762    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:17.733435 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:17.733386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:17.895889 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:17.895815    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d87c34-695d-4482-8df5-0b07ab7378b3" path="/var/lib/kubelet/pods/88d87c34-695d-4482-8df5-0b07ab7378b3/volumes"
Sep 09 20:12:18.691784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:18.691737    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:19.936994 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:19.936965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:20.042092 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:20.042061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.762013 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:23.761981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.921524 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:23.921485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:24.037925 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.037846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:24.122590 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.122557    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:12:24.122941 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.122924    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d87c34-695d-4482-8df5-0b07ab7378b3" containerName="istio-proxy"
Sep 09 20:12:24.123024 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.122944    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d87c34-695d-4482-8df5-0b07ab7378b3" containerName="istio-proxy"
Sep 09 20:12:24.123080 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.123047    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d87c34-695d-4482-8df5-0b07ab7378b3" containerName="istio-proxy"
Sep 09 20:12:24.125092 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.125071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.127747 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.127726    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:12:24.127926 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.127906    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-t6hcr\""
Sep 09 20:12:24.128233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.128216    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 09 20:12:24.129038 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.129015    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 09 20:12:24.129150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.129116    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-kwgcw\""
Sep 09 20:12:24.129225 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.129155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 09 20:12:24.129225 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.129217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 09 20:12:24.141082 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.141061    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:12:24.201659 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.201629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:24.228213 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.228164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.228370 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.228234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.228370 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.228255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61300534-781d-4e59-ac26-2b289ed5a292-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.228370 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.228301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.228487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.228386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.228487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.228410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.228487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.228440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.228487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.228457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zvr\" (UniqueName: \"kubernetes.io/projected/61300534-781d-4e59-ac26-2b289ed5a292-kube-api-access-79zvr\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329170 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329170 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zvr\" (UniqueName: \"kubernetes.io/projected/61300534-781d-4e59-ac26-2b289ed5a292-kube-api-access-79zvr\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329584 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61300534-781d-4e59-ac26-2b289ed5a292-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329584 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329584 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329584 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329795 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.329973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.329951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.330123 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.330106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61300534-781d-4e59-ac26-2b289ed5a292-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.331624 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.331597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.331813 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.331791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.337113 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.337093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zvr\" (UniqueName: \"kubernetes.io/projected/61300534-781d-4e59-ac26-2b289ed5a292-kube-api-access-79zvr\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.436062 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.436015    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:12:24.713018 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.712986    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:12:24.716087 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:12:24.716058    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61300534_781d_4e59_ac26_2b289ed5a292.slice/crio-921024c5f26b067653b27475aabbf788c2fac2d63547e25995c8fb14488d8077 WatchSource:0}: Error finding container 921024c5f26b067653b27475aabbf788c2fac2d63547e25995c8fb14488d8077: Status 404 returned error can't find the container with id 921024c5f26b067653b27475aabbf788c2fac2d63547e25995c8fb14488d8077
Sep 09 20:12:24.718732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:24.718711    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:12:25.000797 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:25.000716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:25.276549 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:25.276456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"61300534-781d-4e59-ac26-2b289ed5a292","Type":"ContainerStarted","Data":"921024c5f26b067653b27475aabbf788c2fac2d63547e25995c8fb14488d8077"}
Sep 09 20:12:26.423959 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:26.423907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:29.299730 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:29.299695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"61300534-781d-4e59-ac26-2b289ed5a292","Type":"ContainerStarted","Data":"69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476"}
Sep 09 20:12:31.310491 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:31.310457    2578 generic.go:356] "Generic (PLEG): container finished" podID="61300534-781d-4e59-ac26-2b289ed5a292" containerID="69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476" exitCode=0
Sep 09 20:12:31.310882 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:31.310532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"61300534-781d-4e59-ac26-2b289ed5a292","Type":"ContainerDied","Data":"69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476"}
Sep 09 20:12:32.060349 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.060236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:32.146057 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.146021    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:32.820489 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.820451    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"]
Sep 09 20:12:32.826449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.826422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:32.834173 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.834144    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-wzxzx\""
Sep 09 20:12:32.834416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.834396    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 09 20:12:32.834501 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.834446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"]
Sep 09 20:12:32.872028 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.871986    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:32.908735 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.908694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:32.908895 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.908788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:32.908895 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.908835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j759g\" (UniqueName: \"kubernetes.io/projected/c9d07542-4a1b-43a8-ac1f-2081bb94330d-kube-api-access-j759g\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:32.908968 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.908906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:32.908968 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.908941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:32.909028 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:32.908969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.010370 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.010332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.010370 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.010386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.010650 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.010418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.010719 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.010691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.011280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.010775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.011280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.010824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j759g\" (UniqueName: \"kubernetes.io/projected/c9d07542-4a1b-43a8-ac1f-2081bb94330d-kube-api-access-j759g\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.011280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.010937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.011721 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.011697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.011832 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.011767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.015119 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.015060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.017453 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.017397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.019894 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.019869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j759g\" (UniqueName: \"kubernetes.io/projected/c9d07542-4a1b-43a8-ac1f-2081bb94330d-kube-api-access-j759g\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:33.143522 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:33.143482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:34.242344 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:34.242240    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"]
Sep 09 20:12:34.337857 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:34.337817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g" event={"ID":"c9d07542-4a1b-43a8-ac1f-2081bb94330d","Type":"ContainerStarted","Data":"ab1dcc02fc4b0aa2dafe6d1e31f8a2466878b40f47b2da76b35cf94f48ee8bb4"}
Sep 09 20:12:35.171458 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:35.170875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:35.269296 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:35.269244    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:35.370357 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:35.370296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g" event={"ID":"c9d07542-4a1b-43a8-ac1f-2081bb94330d","Type":"ContainerStarted","Data":"b1ccc0145b8e56a21c3fa68171a6fd66636d58c29739f478ac63c59580ea8e85"}
Sep 09 20:12:35.388677 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:35.388612    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g" podStartSLOduration=3.388590006 podStartE2EDuration="3.388590006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:35.388409071 +0000 UTC m=+1063.977062731" watchObservedRunningTime="2026-09-09 20:12:35.388590006 +0000 UTC m=+1063.977243664"
Sep 09 20:12:36.161332 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:36.155860    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:40.130706 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:40.130665    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"]
Sep 09 20:12:40.138471 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:40.138441    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"]
Sep 09 20:12:40.138697 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:40.138672    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv" podUID="5649f61c-86b0-440a-9c52-ac868ec5ac45" containerName="istio-proxy" containerID="cri-o://dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a" gracePeriod=30
Sep 09 20:12:40.141137 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:12:40.141068    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:12:40.141137 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:12:40.141137 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:12:40.141137 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="b1ccc0145b8e56a21c3fa68171a6fd66636d58c29739f478ac63c59580ea8e85" cmd=["/bin/sleep","15"]
Sep 09 20:12:40.141137 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:40.141125    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g" message="" err=<
Sep 09 20:12:40.141137 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:12:40.141137 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:12:40.141137 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:12:40.141566 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:12:40.141145    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:12:40.141566 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:12:40.141566 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:12:40.141566 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g" podUID="c9d07542-4a1b-43a8-ac1f-2081bb94330d" containerName="main" containerID="cri-o://b1ccc0145b8e56a21c3fa68171a6fd66636d58c29739f478ac63c59580ea8e85"
Sep 09 20:12:40.141566 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:40.141192    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g" podUID="c9d07542-4a1b-43a8-ac1f-2081bb94330d" containerName="main" containerID="cri-o://b1ccc0145b8e56a21c3fa68171a6fd66636d58c29739f478ac63c59580ea8e85" gracePeriod=60
Sep 09 20:12:40.226276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:40.226242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:42.242537 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:42.242478    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv" podUID="5649f61c-86b0-440a-9c52-ac868ec5ac45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:12:45.437159 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.437122    2578 generic.go:356] "Generic (PLEG): container finished" podID="c9d07542-4a1b-43a8-ac1f-2081bb94330d" containerID="b1ccc0145b8e56a21c3fa68171a6fd66636d58c29739f478ac63c59580ea8e85" exitCode=0
Sep 09 20:12:45.437654 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.437201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g" event={"ID":"c9d07542-4a1b-43a8-ac1f-2081bb94330d","Type":"ContainerDied","Data":"b1ccc0145b8e56a21c3fa68171a6fd66636d58c29739f478ac63c59580ea8e85"}
Sep 09 20:12:45.606093 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.606061    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:45.656881 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.656738    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-dshm\") pod \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") "
Sep 09 20:12:45.656881 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.656792    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tmp-dir\") pod \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") "
Sep 09 20:12:45.656881 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.656843    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tls-certs\") pod \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") "
Sep 09 20:12:45.656881 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.656878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-home\") pod \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") "
Sep 09 20:12:45.657264 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.656930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-model-cache\") pod \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") "
Sep 09 20:12:45.657264 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.656969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j759g\" (UniqueName: \"kubernetes.io/projected/c9d07542-4a1b-43a8-ac1f-2081bb94330d-kube-api-access-j759g\") pod \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\" (UID: \"c9d07542-4a1b-43a8-ac1f-2081bb94330d\") "
Sep 09 20:12:45.658373 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.657770    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9d07542-4a1b-43a8-ac1f-2081bb94330d" (UID: "c9d07542-4a1b-43a8-ac1f-2081bb94330d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:45.658373 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.657959    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-home" (OuterVolumeSpecName: "home") pod "c9d07542-4a1b-43a8-ac1f-2081bb94330d" (UID: "c9d07542-4a1b-43a8-ac1f-2081bb94330d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:45.658373 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.658172    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9d07542-4a1b-43a8-ac1f-2081bb94330d" (UID: "c9d07542-4a1b-43a8-ac1f-2081bb94330d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:45.660820 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.660788    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9d07542-4a1b-43a8-ac1f-2081bb94330d" (UID: "c9d07542-4a1b-43a8-ac1f-2081bb94330d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:45.661735 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.661696    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d07542-4a1b-43a8-ac1f-2081bb94330d-kube-api-access-j759g" (OuterVolumeSpecName: "kube-api-access-j759g") pod "c9d07542-4a1b-43a8-ac1f-2081bb94330d" (UID: "c9d07542-4a1b-43a8-ac1f-2081bb94330d"). InnerVolumeSpecName "kube-api-access-j759g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:45.662213 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.662184    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-dshm" (OuterVolumeSpecName: "dshm") pod "c9d07542-4a1b-43a8-ac1f-2081bb94330d" (UID: "c9d07542-4a1b-43a8-ac1f-2081bb94330d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:45.752665 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.752642    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:45.757929 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.757904    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.757929 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.757931    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.758099 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.757946    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j759g\" (UniqueName: \"kubernetes.io/projected/c9d07542-4a1b-43a8-ac1f-2081bb94330d-kube-api-access-j759g\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.758099 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.757963    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.758099 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.757977    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.758099 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.757991    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d07542-4a1b-43a8-ac1f-2081bb94330d-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.859018 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.858988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-credential-socket\") pod \"5649f61c-86b0-440a-9c52-ac868ec5ac45\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") "
Sep 09 20:12:45.859204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-socket\") pod \"5649f61c-86b0-440a-9c52-ac868ec5ac45\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") "
Sep 09 20:12:45.859204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-podinfo\") pod \"5649f61c-86b0-440a-9c52-ac868ec5ac45\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") "
Sep 09 20:12:45.859204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859087    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-envoy\") pod \"5649f61c-86b0-440a-9c52-ac868ec5ac45\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") "
Sep 09 20:12:45.859204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859138    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-token\") pod \"5649f61c-86b0-440a-9c52-ac868ec5ac45\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") "
Sep 09 20:12:45.859204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859167    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-data\") pod \"5649f61c-86b0-440a-9c52-ac868ec5ac45\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") "
Sep 09 20:12:45.859204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5649f61c-86b0-440a-9c52-ac868ec5ac45-istiod-ca-cert\") pod \"5649f61c-86b0-440a-9c52-ac868ec5ac45\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") "
Sep 09 20:12:45.859552 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-certs\") pod \"5649f61c-86b0-440a-9c52-ac868ec5ac45\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") "
Sep 09 20:12:45.859552 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859274    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x66hr\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-kube-api-access-x66hr\") pod \"5649f61c-86b0-440a-9c52-ac868ec5ac45\" (UID: \"5649f61c-86b0-440a-9c52-ac868ec5ac45\") "
Sep 09 20:12:45.859552 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859339    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5649f61c-86b0-440a-9c52-ac868ec5ac45" (UID: "5649f61c-86b0-440a-9c52-ac868ec5ac45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:45.859552 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859352    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5649f61c-86b0-440a-9c52-ac868ec5ac45" (UID: "5649f61c-86b0-440a-9c52-ac868ec5ac45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:45.859552 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859544    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-credential-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.859812 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859562    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.859812 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5649f61c-86b0-440a-9c52-ac868ec5ac45" (UID: "5649f61c-86b0-440a-9c52-ac868ec5ac45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:45.859812 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859744    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-data" (OuterVolumeSpecName: "istio-data") pod "5649f61c-86b0-440a-9c52-ac868ec5ac45" (UID: "5649f61c-86b0-440a-9c52-ac868ec5ac45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:45.859812 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.859772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5649f61c-86b0-440a-9c52-ac868ec5ac45-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5649f61c-86b0-440a-9c52-ac868ec5ac45" (UID: "5649f61c-86b0-440a-9c52-ac868ec5ac45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:45.861712 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.861689    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5649f61c-86b0-440a-9c52-ac868ec5ac45" (UID: "5649f61c-86b0-440a-9c52-ac868ec5ac45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:45.861839 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.861719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5649f61c-86b0-440a-9c52-ac868ec5ac45" (UID: "5649f61c-86b0-440a-9c52-ac868ec5ac45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:12:45.861904 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.861867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-token" (OuterVolumeSpecName: "istio-token") pod "5649f61c-86b0-440a-9c52-ac868ec5ac45" (UID: "5649f61c-86b0-440a-9c52-ac868ec5ac45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:45.862132 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.862106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-kube-api-access-x66hr" (OuterVolumeSpecName: "kube-api-access-x66hr") pod "5649f61c-86b0-440a-9c52-ac868ec5ac45" (UID: "5649f61c-86b0-440a-9c52-ac868ec5ac45"). InnerVolumeSpecName "kube-api-access-x66hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:45.960411 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.960372    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-podinfo\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.960411 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.960406    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-envoy\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.960411 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.960418    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-token\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.960692 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.960429    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-istio-data\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.960692 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.960442    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5649f61c-86b0-440a-9c52-ac868ec5ac45-istiod-ca-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.960692 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.960453    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5649f61c-86b0-440a-9c52-ac868ec5ac45-workload-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:45.960692 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:45.960464    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66hr\" (UniqueName: \"kubernetes.io/projected/5649f61c-86b0-440a-9c52-ac868ec5ac45-kube-api-access-x66hr\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:12:46.444126 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.444090    2578 generic.go:356] "Generic (PLEG): container finished" podID="5649f61c-86b0-440a-9c52-ac868ec5ac45" containerID="dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a" exitCode=0
Sep 09 20:12:46.444950 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.444148    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"
Sep 09 20:12:46.444950 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.444186    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv" event={"ID":"5649f61c-86b0-440a-9c52-ac868ec5ac45","Type":"ContainerDied","Data":"dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a"}
Sep 09 20:12:46.444950 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.444242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv" event={"ID":"5649f61c-86b0-440a-9c52-ac868ec5ac45","Type":"ContainerDied","Data":"e4a499c37e0288d3cadbb2bb3c9a8d75bc899a64abce46fe02a7c12eb1a3421b"}
Sep 09 20:12:46.444950 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.444270    2578 scope.go:117] "RemoveContainer" containerID="dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a"
Sep 09 20:12:46.446141 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.446075    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"
Sep 09 20:12:46.446141 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.446092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g" event={"ID":"c9d07542-4a1b-43a8-ac1f-2081bb94330d","Type":"ContainerDied","Data":"ab1dcc02fc4b0aa2dafe6d1e31f8a2466878b40f47b2da76b35cf94f48ee8bb4"}
Sep 09 20:12:46.453890 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.453872    2578 scope.go:117] "RemoveContainer" containerID="dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a"
Sep 09 20:12:46.454175 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:12:46.454152    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a\": container with ID starting with dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a not found: ID does not exist" containerID="dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a"
Sep 09 20:12:46.454230 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.454185    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a"} err="failed to get container status \"dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a\": rpc error: code = NotFound desc = could not find container \"dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a\": container with ID starting with dea604236c219f9a4540141408be3060dcbbc8ea8a735537aabc01dab03e452a not found: ID does not exist"
Sep 09 20:12:46.454230 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.454202    2578 scope.go:117] "RemoveContainer" containerID="b1ccc0145b8e56a21c3fa68171a6fd66636d58c29739f478ac63c59580ea8e85"
Sep 09 20:12:46.463184 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.463157    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"]
Sep 09 20:12:46.467593 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.467571    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf74f2g"]
Sep 09 20:12:46.477853 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.477826    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"]
Sep 09 20:12:46.483493 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.483470    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-gsdjv"]
Sep 09 20:12:46.924467 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:46.924429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:47.897711 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:47.897675    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5649f61c-86b0-440a-9c52-ac868ec5ac45" path="/var/lib/kubelet/pods/5649f61c-86b0-440a-9c52-ac868ec5ac45/volumes"
Sep 09 20:12:47.898163 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:47.898148    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9d07542-4a1b-43a8-ac1f-2081bb94330d" path="/var/lib/kubelet/pods/c9d07542-4a1b-43a8-ac1f-2081bb94330d/volumes"
Sep 09 20:12:49.644545 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:12:49.644487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:02.533878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:02.533843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"61300534-781d-4e59-ac26-2b289ed5a292","Type":"ContainerStarted","Data":"219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838"}
Sep 09 20:13:02.554696 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:02.554639    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.09890647 podStartE2EDuration="38.554621251s" totalImagesPullingTime="34.30987334s" totalInitContainerRuntime="2.145841441s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:02.553854209 +0000 UTC m=+1091.142507890" watchObservedRunningTime="2026-09-09 20:13:02.554621251 +0000 UTC m=+1091.143274909"
Sep 09 20:13:12.758414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.758377    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"]
Sep 09 20:13:12.759468 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.759441    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5649f61c-86b0-440a-9c52-ac868ec5ac45" containerName="istio-proxy"
Sep 09 20:13:12.759468 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.759467    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5649f61c-86b0-440a-9c52-ac868ec5ac45" containerName="istio-proxy"
Sep 09 20:13:12.759664 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.759507    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d07542-4a1b-43a8-ac1f-2081bb94330d" containerName="main"
Sep 09 20:13:12.759664 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.759517    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d07542-4a1b-43a8-ac1f-2081bb94330d" containerName="main"
Sep 09 20:13:12.759664 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.759606    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5649f61c-86b0-440a-9c52-ac868ec5ac45" containerName="istio-proxy"
Sep 09 20:13:12.759664 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.759620    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d07542-4a1b-43a8-ac1f-2081bb94330d" containerName="main"
Sep 09 20:13:12.782813 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.782778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"]
Sep 09 20:13:12.782998 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.782921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:12.785474 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.785450    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 09 20:13:12.785722 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.785701    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-td82v\""
Sep 09 20:13:12.785870 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.785765    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 09 20:13:12.785944 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.785924    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:13:12.786271 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.786245    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 09 20:13:12.786563 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.786526    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 09 20:13:12.787551 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.787528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:12.905080 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.905045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tls-certs\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:12.905289 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.905111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-home\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:12.905289 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.905190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpdx\" (UniqueName: \"kubernetes.io/projected/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kube-api-access-sfpdx\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:12.905289 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.905245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-model-cache\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:12.905507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.905295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-dshm\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:12.905507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.905328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tmp-dir\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:12.905507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.905352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-cabundle-cert\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:12.905507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:12.905399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kserve-provision-location\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.006548 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.006509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpdx\" (UniqueName: \"kubernetes.io/projected/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kube-api-access-sfpdx\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.006743 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.006567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-model-cache\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.006743 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.006590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-dshm\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.006743 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.006612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tmp-dir\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.006743 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.006637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-cabundle-cert\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.006967 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.006924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kserve-provision-location\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.007029 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.006979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tls-certs\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.007084 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.007026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tmp-dir\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.007084 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.007047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-model-cache\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.007185 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.007101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-home\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.007481 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.007452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kserve-provision-location\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.007481 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.007476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-home\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.007712 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.007675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-cabundle-cert\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.009680 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.009624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tls-certs\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.009813 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.009791    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-dshm\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.016074 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.016047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpdx\" (UniqueName: \"kubernetes.io/projected/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kube-api-access-sfpdx\") pod \"auth-enabled-test-kserve-b5cfc6f64-cpxf4\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.048527 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.048489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:13.095715 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.095680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:13.416217 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.416180    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"]
Sep 09 20:13:13.419467 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:13:13.419443    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4f2cdc_3f93_4aae_8f55_72bf49989bbc.slice/crio-0ff2e8c92df3139552b07bfc9e0d1014d394069bff8c9196603ee9a3cb6a1a97 WatchSource:0}: Error finding container 0ff2e8c92df3139552b07bfc9e0d1014d394069bff8c9196603ee9a3cb6a1a97: Status 404 returned error can't find the container with id 0ff2e8c92df3139552b07bfc9e0d1014d394069bff8c9196603ee9a3cb6a1a97
Sep 09 20:13:13.585901 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.585859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" event={"ID":"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc","Type":"ContainerStarted","Data":"1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b"}
Sep 09 20:13:13.586082 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:13.585986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" event={"ID":"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc","Type":"ContainerStarted","Data":"0ff2e8c92df3139552b07bfc9e0d1014d394069bff8c9196603ee9a3cb6a1a97"}
Sep 09 20:13:16.604166 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:16.604132    2578 generic.go:356] "Generic (PLEG): container finished" podID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerID="1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b" exitCode=0
Sep 09 20:13:16.604578 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:16.604200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" event={"ID":"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc","Type":"ContainerDied","Data":"1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b"}
Sep 09 20:13:16.952548 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:16.952456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:17.488725 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:17.488683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:17.612811 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:17.612771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" event={"ID":"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc","Type":"ContainerStarted","Data":"ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354"}
Sep 09 20:13:17.646135 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:17.646074    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podStartSLOduration=3.190579854 podStartE2EDuration="5.646051523s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.455471669s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:17.63507786 +0000 UTC m=+1106.223731518" watchObservedRunningTime="2026-09-09 20:13:17.646051523 +0000 UTC m=+1106.234705182"
Sep 09 20:13:18.096395 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:18.096351    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:19.918009 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:19.917973    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:23.096463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:23.096419    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:13:23.098243 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:23.098206    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:13:23.966450 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:23.966415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:28.338927 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:28.338888    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:31.668280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:31.668247    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:31.733528 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:31.733485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:33.096676 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:33.096628    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:13:43.096442 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:43.096391    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:13:53.096931 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:13:53.096877    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:14:03.096257 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:03.096199    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:14:13.096138 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:13.096089    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:14:20.681336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:20.681231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:20.749257 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:20.749210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:23.097157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:23.097110    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:14:30.890646 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:30.890599    2578 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-t6hcr\" not found"
Sep 09 20:14:31.001682 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:31.001436    2578 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:31.001682 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:31.001523    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs podName:61300534-781d-4e59-ac26-2b289ed5a292 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:31.501499974 +0000 UTC m=+1180.090153640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "61300534-781d-4e59-ac26-2b289ed5a292") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:31.506998 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:31.506957    2578 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:31.507185 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:31.507031    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs podName:61300534-781d-4e59-ac26-2b289ed5a292 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:32.507016401 +0000 UTC m=+1181.095670036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "61300534-781d-4e59-ac26-2b289ed5a292") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:32.517605 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:32.517561    2578 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:32.518112 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:32.517648    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs podName:61300534-781d-4e59-ac26-2b289ed5a292 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:34.51763165 +0000 UTC m=+1183.106285298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "61300534-781d-4e59-ac26-2b289ed5a292") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:33.096762 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:33.096712    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:14:34.537870 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:34.537834    2578 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:34.538272 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:34.537914    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs podName:61300534-781d-4e59-ac26-2b289ed5a292 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:38.5379 +0000 UTC m=+1187.126553635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "61300534-781d-4e59-ac26-2b289ed5a292") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:36.918897 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:36.918864    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:14:38.578342 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:38.578274    2578 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:38.578854 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:38.578429    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs podName:61300534-781d-4e59-ac26-2b289ed5a292 nodeName:}" failed. No retries permitted until 2026-09-09 20:14:46.57838406 +0000 UTC m=+1195.167037700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "61300534-781d-4e59-ac26-2b289ed5a292") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:43.096619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:43.096573    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:14:46.655392 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:46.655353    2578 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:46.655866 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:46.655426    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs podName:61300534-781d-4e59-ac26-2b289ed5a292 nodeName:}" failed. No retries permitted until 2026-09-09 20:15:02.65541074 +0000 UTC m=+1211.244064375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "61300534-781d-4e59-ac26-2b289ed5a292") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 09 20:14:51.904791 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:51.904759    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:14:51.905257 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:51.904839    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:14:51.941641 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:51.941569    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="61300534-781d-4e59-ac26-2b289ed5a292" containerName="main" containerID="cri-o://219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838" gracePeriod=45
Sep 09 20:14:52.888123 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:52.888094    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:14:53.015229 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.015143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-home\") pod \"61300534-781d-4e59-ac26-2b289ed5a292\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") "
Sep 09 20:14:53.015229 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.015196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-dshm\") pod \"61300534-781d-4e59-ac26-2b289ed5a292\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") "
Sep 09 20:14:53.015766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.015260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79zvr\" (UniqueName: \"kubernetes.io/projected/61300534-781d-4e59-ac26-2b289ed5a292-kube-api-access-79zvr\") pod \"61300534-781d-4e59-ac26-2b289ed5a292\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") "
Sep 09 20:14:53.015766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.015287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-kserve-provision-location\") pod \"61300534-781d-4e59-ac26-2b289ed5a292\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") "
Sep 09 20:14:53.015766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.015373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61300534-781d-4e59-ac26-2b289ed5a292-cabundle-cert\") pod \"61300534-781d-4e59-ac26-2b289ed5a292\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") "
Sep 09 20:14:53.015766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.015406    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-tmp-dir\") pod \"61300534-781d-4e59-ac26-2b289ed5a292\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") "
Sep 09 20:14:53.015766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.015435    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs\") pod \"61300534-781d-4e59-ac26-2b289ed5a292\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") "
Sep 09 20:14:53.015766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.015456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-model-cache\") pod \"61300534-781d-4e59-ac26-2b289ed5a292\" (UID: \"61300534-781d-4e59-ac26-2b289ed5a292\") "
Sep 09 20:14:53.016076 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.016033    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-home" (OuterVolumeSpecName: "home") pod "61300534-781d-4e59-ac26-2b289ed5a292" (UID: "61300534-781d-4e59-ac26-2b289ed5a292"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:53.016210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.016182    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-model-cache" (OuterVolumeSpecName: "model-cache") pod "61300534-781d-4e59-ac26-2b289ed5a292" (UID: "61300534-781d-4e59-ac26-2b289ed5a292"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:53.016653 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.016613    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61300534-781d-4e59-ac26-2b289ed5a292-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "61300534-781d-4e59-ac26-2b289ed5a292" (UID: "61300534-781d-4e59-ac26-2b289ed5a292"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:53.019084 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.019040    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-dshm" (OuterVolumeSpecName: "dshm") pod "61300534-781d-4e59-ac26-2b289ed5a292" (UID: "61300534-781d-4e59-ac26-2b289ed5a292"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:53.019084 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.019059    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61300534-781d-4e59-ac26-2b289ed5a292-kube-api-access-79zvr" (OuterVolumeSpecName: "kube-api-access-79zvr") pod "61300534-781d-4e59-ac26-2b289ed5a292" (UID: "61300534-781d-4e59-ac26-2b289ed5a292"). InnerVolumeSpecName "kube-api-access-79zvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:53.019084 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.019059    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61300534-781d-4e59-ac26-2b289ed5a292" (UID: "61300534-781d-4e59-ac26-2b289ed5a292"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:53.028462 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.028431    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61300534-781d-4e59-ac26-2b289ed5a292" (UID: "61300534-781d-4e59-ac26-2b289ed5a292"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:53.049343 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.049281    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61300534-781d-4e59-ac26-2b289ed5a292" (UID: "61300534-781d-4e59-ac26-2b289ed5a292"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:14:53.096474 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.096420    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:14:53.103709 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.103679    2578 generic.go:356] "Generic (PLEG): container finished" podID="61300534-781d-4e59-ac26-2b289ed5a292" containerID="219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838" exitCode=0
Sep 09 20:14:53.103874 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.103746    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 09 20:14:53.103874 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.103746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"61300534-781d-4e59-ac26-2b289ed5a292","Type":"ContainerDied","Data":"219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838"}
Sep 09 20:14:53.103874 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.103869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"61300534-781d-4e59-ac26-2b289ed5a292","Type":"ContainerDied","Data":"921024c5f26b067653b27475aabbf788c2fac2d63547e25995c8fb14488d8077"}
Sep 09 20:14:53.104026 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.103894    2578 scope.go:117] "RemoveContainer" containerID="219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838"
Sep 09 20:14:53.113425 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.113406    2578 scope.go:117] "RemoveContainer" containerID="69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476"
Sep 09 20:14:53.116291 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.116269    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.116291 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.116289    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.116480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.116298    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79zvr\" (UniqueName: \"kubernetes.io/projected/61300534-781d-4e59-ac26-2b289ed5a292-kube-api-access-79zvr\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.116480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.116337    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.116480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.116348    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61300534-781d-4e59-ac26-2b289ed5a292-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.116480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.116357    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.116480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.116365    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61300534-781d-4e59-ac26-2b289ed5a292-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.116480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.116372    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61300534-781d-4e59-ac26-2b289ed5a292-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:14:53.121750 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.121726    2578 scope.go:117] "RemoveContainer" containerID="219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838"
Sep 09 20:14:53.122034 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:53.122013    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838\": container with ID starting with 219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838 not found: ID does not exist" containerID="219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838"
Sep 09 20:14:53.122113 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.122046    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838"} err="failed to get container status \"219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838\": rpc error: code = NotFound desc = could not find container \"219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838\": container with ID starting with 219c94167d1a3202b79130255adc1b9e9389304b9d86c42fef9c64e4848c9838 not found: ID does not exist"
Sep 09 20:14:53.122113 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.122069    2578 scope.go:117] "RemoveContainer" containerID="69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476"
Sep 09 20:14:53.122439 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:14:53.122411    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476\": container with ID starting with 69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476 not found: ID does not exist" containerID="69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476"
Sep 09 20:14:53.122499 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.122449    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476"} err="failed to get container status \"69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476\": rpc error: code = NotFound desc = could not find container \"69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476\": container with ID starting with 69cdadd07867687fc08a7fa32bc5aebfa8b1eb39f516d3d2ebcbfd686899b476 not found: ID does not exist"
Sep 09 20:14:53.127751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.127726    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:14:53.132812 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.132788    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 09 20:14:53.896482 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:14:53.896448    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61300534-781d-4e59-ac26-2b289ed5a292" path="/var/lib/kubelet/pods/61300534-781d-4e59-ac26-2b289ed5a292/volumes"
Sep 09 20:15:03.096558 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:03.096512    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:15:10.820948 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:10.820887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:10.967411 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:10.967364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:13.096884 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:13.096827    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:15:15.921845 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:15.921814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:16.211497 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:16.211344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:17.735546 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:17.735508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:23.105738 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:23.105708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:15:23.113385 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:23.113360    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:15:23.273822 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:23.273791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:23.341811 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:23.341778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:26.080094 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:26.080059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:32.588033 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:32.587998    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"]
Sep 09 20:15:32.591185 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:32.591158    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:32.658540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:32.658506    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:35.774074 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:35.774027    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:35.835644 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:35.835616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:47.611300 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:47.611252    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" containerID="cri-o://ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354" gracePeriod=45
Sep 09 20:15:48.107223 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.107189    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 09 20:15:48.263422 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.263396    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:15:48.364779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.364690    2578 generic.go:356] "Generic (PLEG): container finished" podID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerID="ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354" exitCode=0
Sep 09 20:15:48.364779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.364741    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"
Sep 09 20:15:48.365022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.364777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" event={"ID":"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc","Type":"ContainerDied","Data":"ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354"}
Sep 09 20:15:48.365022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.364840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4" event={"ID":"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc","Type":"ContainerDied","Data":"0ff2e8c92df3139552b07bfc9e0d1014d394069bff8c9196603ee9a3cb6a1a97"}
Sep 09 20:15:48.365022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.364871    2578 scope.go:117] "RemoveContainer" containerID="ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354"
Sep 09 20:15:48.373019 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.373001    2578 scope.go:117] "RemoveContainer" containerID="1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b"
Sep 09 20:15:48.380210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.380191    2578 scope.go:117] "RemoveContainer" containerID="ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354"
Sep 09 20:15:48.380553 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:15:48.380534    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354\": container with ID starting with ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354 not found: ID does not exist" containerID="ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354"
Sep 09 20:15:48.380613 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.380561    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354"} err="failed to get container status \"ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354\": rpc error: code = NotFound desc = could not find container \"ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354\": container with ID starting with ed603c3604d4885ac55281a4b81d4dd6639ddf6bdc8d29b0e357c56f28ce4354 not found: ID does not exist"
Sep 09 20:15:48.380613 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.380578    2578 scope.go:117] "RemoveContainer" containerID="1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b"
Sep 09 20:15:48.380855 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:15:48.380835    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b\": container with ID starting with 1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b not found: ID does not exist" containerID="1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b"
Sep 09 20:15:48.380920 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.380862    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b"} err="failed to get container status \"1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b\": rpc error: code = NotFound desc = could not find container \"1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b\": container with ID starting with 1ef867d94885d7be8c13a2dcfdb79d1593054bbadfc673883292f22d62733a8b not found: ID does not exist"
Sep 09 20:15:48.387102 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-home\") pod \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") "
Sep 09 20:15:48.387155 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387112    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-cabundle-cert\") pod \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") "
Sep 09 20:15:48.387155 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387138    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kserve-provision-location\") pod \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") "
Sep 09 20:15:48.387267 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387156    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfpdx\" (UniqueName: \"kubernetes.io/projected/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kube-api-access-sfpdx\") pod \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") "
Sep 09 20:15:48.387267 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-dshm\") pod \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") "
Sep 09 20:15:48.387267 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-model-cache\") pod \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") "
Sep 09 20:15:48.387453 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tls-certs\") pod \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") "
Sep 09 20:15:48.387453 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387359    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tmp-dir\") pod \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\" (UID: \"ad4f2cdc-3f93-4aae-8f55-72bf49989bbc\") "
Sep 09 20:15:48.387557 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387492    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" (UID: "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:15:48.387762 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387623    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:15:48.387863 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387840    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" (UID: "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:48.387946 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.387925    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-home" (OuterVolumeSpecName: "home") pod "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" (UID: "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:48.389541 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.389512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kube-api-access-sfpdx" (OuterVolumeSpecName: "kube-api-access-sfpdx") pod "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" (UID: "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc"). InnerVolumeSpecName "kube-api-access-sfpdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:15:48.389761 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.389743    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" (UID: "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:15:48.389901 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.389870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-dshm" (OuterVolumeSpecName: "dshm") pod "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" (UID: "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:48.402382 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.402301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" (UID: "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:48.410521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.410492    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" (UID: "ad4f2cdc-3f93-4aae-8f55-72bf49989bbc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:48.488936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.488901    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:15:48.488936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.488931    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:15:48.488936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.488939    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:15:48.488936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.488947    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:15:48.489200 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.488956    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:15:48.489200 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.488966    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfpdx\" (UniqueName: \"kubernetes.io/projected/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-kube-api-access-sfpdx\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:15:48.489200 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.488975    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:15:48.689548 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.689517    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"]
Sep 09 20:15:48.694510 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:48.694487    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-b5cfc6f64-cpxf4"]
Sep 09 20:15:49.897864 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:49.897826    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" path="/var/lib/kubelet/pods/ad4f2cdc-3f93-4aae-8f55-72bf49989bbc/volumes"
Sep 09 20:15:50.614741 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:50.614695    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:50.686749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:15:50.686712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:22.755825 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:22.755784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:24.153646 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:24.153605    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:29.290779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:29.290745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:35.361372 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361339    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"]
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361663    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main"
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361675    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main"
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361682    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61300534-781d-4e59-ac26-2b289ed5a292" containerName="main"
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361689    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61300534-781d-4e59-ac26-2b289ed5a292" containerName="main"
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361700    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61300534-781d-4e59-ac26-2b289ed5a292" containerName="storage-initializer"
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361706    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61300534-781d-4e59-ac26-2b289ed5a292" containerName="storage-initializer"
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361726    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="storage-initializer"
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361732    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="storage-initializer"
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361779    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="61300534-781d-4e59-ac26-2b289ed5a292" containerName="main"
Sep 09 20:16:35.361872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.361786    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad4f2cdc-3f93-4aae-8f55-72bf49989bbc" containerName="main"
Sep 09 20:16:35.364768 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.364751    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.367988 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.367966    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:16:35.368319 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.368288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 09 20:16:35.368400 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.368335    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 09 20:16:35.369034 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.369005    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-9xpg9\""
Sep 09 20:16:35.369161 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.369065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 09 20:16:35.369161 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.369067    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 09 20:16:35.378056 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.378037    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"]
Sep 09 20:16:35.393875 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.393843    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:35.457130 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.457095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-dshm\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.457301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.457138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-home\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.457301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.457159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.457301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.457183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-model-cache\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.457301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.457255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tmp-dir\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.457301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.457300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b18858a-8e7a-4ea3-b8a0-2745ac605215-cabundle-cert\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.457500 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.457356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tls-certs\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.457500 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.457381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lljfv\" (UniqueName: \"kubernetes.io/projected/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kube-api-access-lljfv\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.500938 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.500907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:35.558833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.558794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tls-certs\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.558833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.558838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lljfv\" (UniqueName: \"kubernetes.io/projected/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kube-api-access-lljfv\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559043 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.558958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-dshm\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559043 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-home\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559132 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559132 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-model-cache\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559132 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tmp-dir\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559260 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b18858a-8e7a-4ea3-b8a0-2745ac605215-cabundle-cert\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-model-cache\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-home\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559701 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tmp-dir\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.559841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.559825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b18858a-8e7a-4ea3-b8a0-2745ac605215-cabundle-cert\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.561366 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.561345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-dshm\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.561480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.561464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tls-certs\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.567999 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.567973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lljfv\" (UniqueName: \"kubernetes.io/projected/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kube-api-access-lljfv\") pod \"auth-invalid-token-test-kserve-c58c9d897-mbj6n\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.611896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.611814    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:35.675099 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.675059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:35.706505 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.706460    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:35.956648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:35.956622    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"]
Sep 09 20:16:35.959299 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:16:35.959270    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b18858a_8e7a_4ea3_b8a0_2745ac605215.slice/crio-77e2a449b89338b6cae8fd0e0992acc7f7f5e922b045f19cde4e2988bedc8ea6 WatchSource:0}: Error finding container 77e2a449b89338b6cae8fd0e0992acc7f7f5e922b045f19cde4e2988bedc8ea6: Status 404 returned error can't find the container with id 77e2a449b89338b6cae8fd0e0992acc7f7f5e922b045f19cde4e2988bedc8ea6
Sep 09 20:16:36.580937 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:36.580898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" event={"ID":"9b18858a-8e7a-4ea3-b8a0-2745ac605215","Type":"ContainerStarted","Data":"4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c"}
Sep 09 20:16:36.581295 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:36.580956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" event={"ID":"9b18858a-8e7a-4ea3-b8a0-2745ac605215","Type":"ContainerStarted","Data":"77e2a449b89338b6cae8fd0e0992acc7f7f5e922b045f19cde4e2988bedc8ea6"}
Sep 09 20:16:38.592379 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:38.592344    2578 generic.go:356] "Generic (PLEG): container finished" podID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerID="4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c" exitCode=0
Sep 09 20:16:38.592755 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:38.592395    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" event={"ID":"9b18858a-8e7a-4ea3-b8a0-2745ac605215","Type":"ContainerDied","Data":"4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c"}
Sep 09 20:16:39.599398 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:39.599354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" event={"ID":"9b18858a-8e7a-4ea3-b8a0-2745ac605215","Type":"ContainerStarted","Data":"472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83"}
Sep 09 20:16:39.625080 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:39.624994    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podStartSLOduration=2.441207941 podStartE2EDuration="4.624978156s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.183770215s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:39.623656637 +0000 UTC m=+1308.212310293" watchObservedRunningTime="2026-09-09 20:16:39.624978156 +0000 UTC m=+1308.213631812"
Sep 09 20:16:39.675858 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:39.675826    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:39.736580 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:39.736547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:39.828215 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:39.828155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:45.675576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:45.675538    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:16:45.677361 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:45.677328    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:16:46.133378 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:46.133347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:48.049530 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:48.049494    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:52.129650 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:52.129617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:52.230020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:52.229989    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:53.525846 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:53.525809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:53.596792 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:53.596756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:55.675864 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:55.675819    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:16:56.182695 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:56.182663    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:56.484247 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:16:56.484150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:03.136769 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:03.136734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:05.676329 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:05.676270    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:17:06.148477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:06.148437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:15.675601 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:15.675493    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:17:25.676173 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:25.676129    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:17:30.756647 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:30.756595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:30.889356 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:30.889296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:35.676173 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:35.676130    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:17:45.675573 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:45.675524    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:17:55.676387 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:17:55.676342    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:18:05.675830 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:05.675776    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:18:09.668373 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:09.668337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:15.675673 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:15.675625    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:18:20.360663 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:20.360620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:20.535180 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:20.535136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:25.676055 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:25.676007    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
Sep 09 20:18:26.766604 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:26.766559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:32.636021 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:32.635983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:32.740433 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:32.740392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:33.070911 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:33.070816    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:33.323148 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:33.323051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:35.692271 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:35.692230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:18:35.700386 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:35.700360    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:18:42.628615 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:42.628577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:46.979530 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:46.979441    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"]
Sep 09 20:18:46.980417 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:46.980384    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:18:47.069633 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:18:47.069592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:02.001924 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.001867    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" containerID="cri-o://472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83" gracePeriod=45
Sep 09 20:19:02.750844 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.750816    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:19:02.838318 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.838216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kserve-provision-location\") pod \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") "
Sep 09 20:19:02.838318 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.838260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tmp-dir\") pod \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") "
Sep 09 20:19:02.838318 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.838289    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lljfv\" (UniqueName: \"kubernetes.io/projected/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kube-api-access-lljfv\") pod \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") "
Sep 09 20:19:02.838602 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.838357    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b18858a-8e7a-4ea3-b8a0-2745ac605215-cabundle-cert\") pod \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") "
Sep 09 20:19:02.838602 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.838377    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-home\") pod \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") "
Sep 09 20:19:02.838602 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.838393    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tls-certs\") pod \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") "
Sep 09 20:19:02.838602 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.838417    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-dshm\") pod \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") "
Sep 09 20:19:02.838816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.838751    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-model-cache\") pod \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\" (UID: \"9b18858a-8e7a-4ea3-b8a0-2745ac605215\") "
Sep 09 20:19:02.838867 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.838807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b18858a-8e7a-4ea3-b8a0-2745ac605215-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b18858a-8e7a-4ea3-b8a0-2745ac605215" (UID: "9b18858a-8e7a-4ea3-b8a0-2745ac605215"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:19:02.839093 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.839042    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b18858a-8e7a-4ea3-b8a0-2745ac605215-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:19:02.839282 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.839258    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b18858a-8e7a-4ea3-b8a0-2745ac605215" (UID: "9b18858a-8e7a-4ea3-b8a0-2745ac605215"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:02.839469 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.839444    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-home" (OuterVolumeSpecName: "home") pod "9b18858a-8e7a-4ea3-b8a0-2745ac605215" (UID: "9b18858a-8e7a-4ea3-b8a0-2745ac605215"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:02.840979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.840951    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-dshm" (OuterVolumeSpecName: "dshm") pod "9b18858a-8e7a-4ea3-b8a0-2745ac605215" (UID: "9b18858a-8e7a-4ea3-b8a0-2745ac605215"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:02.841158 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.841131    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b18858a-8e7a-4ea3-b8a0-2745ac605215" (UID: "9b18858a-8e7a-4ea3-b8a0-2745ac605215"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:19:02.841228 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.841141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kube-api-access-lljfv" (OuterVolumeSpecName: "kube-api-access-lljfv") pod "9b18858a-8e7a-4ea3-b8a0-2745ac605215" (UID: "9b18858a-8e7a-4ea3-b8a0-2745ac605215"). InnerVolumeSpecName "kube-api-access-lljfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:02.856855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.856826    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b18858a-8e7a-4ea3-b8a0-2745ac605215" (UID: "9b18858a-8e7a-4ea3-b8a0-2745ac605215"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:02.868788 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.868761    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b18858a-8e7a-4ea3-b8a0-2745ac605215" (UID: "9b18858a-8e7a-4ea3-b8a0-2745ac605215"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:02.940143 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.940114    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:19:02.940143 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.940140    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:19:02.940143 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.940149    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:19:02.940348 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.940156    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:19:02.940348 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.940164    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:19:02.940348 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.940173    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b18858a-8e7a-4ea3-b8a0-2745ac605215-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:19:02.940348 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:02.940185    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lljfv\" (UniqueName: \"kubernetes.io/projected/9b18858a-8e7a-4ea3-b8a0-2745ac605215-kube-api-access-lljfv\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:19:03.263934 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.263895    2578 generic.go:356] "Generic (PLEG): container finished" podID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerID="472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83" exitCode=0
Sep 09 20:19:03.264325 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.263947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" event={"ID":"9b18858a-8e7a-4ea3-b8a0-2745ac605215","Type":"ContainerDied","Data":"472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83"}
Sep 09 20:19:03.264325 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.263993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" event={"ID":"9b18858a-8e7a-4ea3-b8a0-2745ac605215","Type":"ContainerDied","Data":"77e2a449b89338b6cae8fd0e0992acc7f7f5e922b045f19cde4e2988bedc8ea6"}
Sep 09 20:19:03.264325 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.264014    2578 scope.go:117] "RemoveContainer" containerID="472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83"
Sep 09 20:19:03.264325 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.263951    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"
Sep 09 20:19:03.272507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.272484    2578 scope.go:117] "RemoveContainer" containerID="4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c"
Sep 09 20:19:03.279717 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.279701    2578 scope.go:117] "RemoveContainer" containerID="472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83"
Sep 09 20:19:03.279937 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:19:03.279919    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83\": container with ID starting with 472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83 not found: ID does not exist" containerID="472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83"
Sep 09 20:19:03.279987 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.279944    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83"} err="failed to get container status \"472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83\": rpc error: code = NotFound desc = could not find container \"472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83\": container with ID starting with 472ff881235d6f5c92054926592742589d969abb441b40344a6319b519d7ce83 not found: ID does not exist"
Sep 09 20:19:03.279987 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.279958    2578 scope.go:117] "RemoveContainer" containerID="4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c"
Sep 09 20:19:03.280168 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:19:03.280149    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c\": container with ID starting with 4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c not found: ID does not exist" containerID="4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c"
Sep 09 20:19:03.280209 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.280175    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c"} err="failed to get container status \"4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c\": rpc error: code = NotFound desc = could not find container \"4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c\": container with ID starting with 4abd6202a861fdc1a2dbb1580aad9355559569ca6aa6f37d118171a8fe2ab03c not found: ID does not exist"
Sep 09 20:19:03.289259 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.289234    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"]
Sep 09 20:19:03.293219 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.293196    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n"]
Sep 09 20:19:03.692751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.692704    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-c58c9d897-mbj6n" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": context deadline exceeded"
Sep 09 20:19:03.896181 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:03.896133    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" path="/var/lib/kubelet/pods/9b18858a-8e7a-4ea3-b8a0-2745ac605215/volumes"
Sep 09 20:19:07.160163 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:07.160127    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:07.237644 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:07.237605    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:35.854008 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:35.853969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:35.949194 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:35.949165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:42.489088 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:42.489026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:42.598215 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:42.598180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:42.749417 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:42.749337    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:48.383040 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:48.383001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:48.465597 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:48.465557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:51.931481 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:51.931451    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:19:51.931931 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:51.931912    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:19:57.995668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:57.995628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:58.097819 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:19:58.097785    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:01.591495 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:01.591465    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:03.518538 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:03.518494    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:09.052830 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:09.052794    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:09.124699 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:09.124663    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:13.833498 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.833464    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"]
Sep 09 20:20:13.833930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.833773    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="storage-initializer"
Sep 09 20:20:13.833930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.833784    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="storage-initializer"
Sep 09 20:20:13.833930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.833799    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main"
Sep 09 20:20:13.833930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.833804    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main"
Sep 09 20:20:13.833930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.833865    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b18858a-8e7a-4ea3-b8a0-2745ac605215" containerName="main"
Sep 09 20:20:13.837915 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.837893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:13.841603 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.841570    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 09 20:20:13.841741 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.841624    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 09 20:20:13.841741 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.841662    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 09 20:20:13.841741 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.841662    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-7kvrs\""
Sep 09 20:20:13.846567 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.846546    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"]
Sep 09 20:20:13.862080 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.862059    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:13.932637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.932605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:13.932776 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.932655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:13.932776 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.932728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:13.932776 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.932756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvhk\" (UniqueName: \"kubernetes.io/projected/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-kube-api-access-rgvhk\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:13.932930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.932848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:13.932930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:13.932884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.033798 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.033718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.033798 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.033758    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.033798 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.033786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.034087 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.033826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.034087 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.033855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.034087 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.033879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvhk\" (UniqueName: \"kubernetes.io/projected/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-kube-api-access-rgvhk\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.034349 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.034154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.034349 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.034336    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.034440 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.034355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.036066 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.036044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.036374 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.036356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.045247 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.045228    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvhk\" (UniqueName: \"kubernetes.io/projected/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-kube-api-access-rgvhk\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k59dm\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.069491 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.069463    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"]
Sep 09 20:20:14.073360 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.073340    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:14.075924 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.075907    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-m5n7v\""
Sep 09 20:20:14.085045 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.084999    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"]
Sep 09 20:20:14.118419 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.118391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:14.134709 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.134680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d0e4d2-008a-4833-9771-9a95033170a6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd\" (UID: \"e9d0e4d2-008a-4833-9771-9a95033170a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:14.134825 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.134768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6nw\" (UniqueName: \"kubernetes.io/projected/e9d0e4d2-008a-4833-9771-9a95033170a6-kube-api-access-tp6nw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd\" (UID: \"e9d0e4d2-008a-4833-9771-9a95033170a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:14.149656 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.149635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:14.236134 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.236095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6nw\" (UniqueName: \"kubernetes.io/projected/e9d0e4d2-008a-4833-9771-9a95033170a6-kube-api-access-tp6nw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd\" (UID: \"e9d0e4d2-008a-4833-9771-9a95033170a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:14.236323 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.236174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d0e4d2-008a-4833-9771-9a95033170a6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd\" (UID: \"e9d0e4d2-008a-4833-9771-9a95033170a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:14.238610 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.238586    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d0e4d2-008a-4833-9771-9a95033170a6-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd\" (UID: \"e9d0e4d2-008a-4833-9771-9a95033170a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:14.244920 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.244896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6nw\" (UniqueName: \"kubernetes.io/projected/e9d0e4d2-008a-4833-9771-9a95033170a6-kube-api-access-tp6nw\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd\" (UID: \"e9d0e4d2-008a-4833-9771-9a95033170a6\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:14.384946 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.384917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:14.426921 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.426889    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"]
Sep 09 20:20:14.429086 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:20:14.429058    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda34ca2f_a624_4d1a_a745_4f4cbd4da09d.slice/crio-0bc56cb06ca0be74e96a1a36fc7245313371a126174dca4589069d1afa3de76b WatchSource:0}: Error finding container 0bc56cb06ca0be74e96a1a36fc7245313371a126174dca4589069d1afa3de76b: Status 404 returned error can't find the container with id 0bc56cb06ca0be74e96a1a36fc7245313371a126174dca4589069d1afa3de76b
Sep 09 20:20:14.582885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.582843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm" event={"ID":"da34ca2f-a624-4d1a-a745-4f4cbd4da09d","Type":"ContainerStarted","Data":"f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0"}
Sep 09 20:20:14.583080 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.582914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm" event={"ID":"da34ca2f-a624-4d1a-a745-4f4cbd4da09d","Type":"ContainerStarted","Data":"0bc56cb06ca0be74e96a1a36fc7245313371a126174dca4589069d1afa3de76b"}
Sep 09 20:20:14.614988 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.614927    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm" podStartSLOduration=1.614910508 podStartE2EDuration="1.614910508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:14.614579089 +0000 UTC m=+1523.203232746" watchObservedRunningTime="2026-09-09 20:20:14.614910508 +0000 UTC m=+1523.203564216"
Sep 09 20:20:14.662268 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.662186    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"]
Sep 09 20:20:14.666050 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:20:14.666016    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d0e4d2_008a_4833_9771_9a95033170a6.slice/crio-57fcf4859d0148bcb015c41dc031440bc978df3f7df68357a0a73d85d33e7c6f WatchSource:0}: Error finding container 57fcf4859d0148bcb015c41dc031440bc978df3f7df68357a0a73d85d33e7c6f: Status 404 returned error can't find the container with id 57fcf4859d0148bcb015c41dc031440bc978df3f7df68357a0a73d85d33e7c6f
Sep 09 20:20:14.668549 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:14.668525    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:20:15.150004 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:15.149962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:15.590914 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:15.590792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd" event={"ID":"e9d0e4d2-008a-4833-9771-9a95033170a6","Type":"ContainerStarted","Data":"57fcf4859d0148bcb015c41dc031440bc978df3f7df68357a0a73d85d33e7c6f"}
Sep 09 20:20:17.600872 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:17.600837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd" event={"ID":"e9d0e4d2-008a-4833-9771-9a95033170a6","Type":"ContainerStarted","Data":"18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564"}
Sep 09 20:20:17.601382 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:17.600909    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:17.619704 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:17.619650    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd" podStartSLOduration=0.995535922 podStartE2EDuration="3.619631942s" totalImagesPullingTime="2.62409602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:17.619238162 +0000 UTC m=+1526.207891820" watchObservedRunningTime="2026-09-09 20:20:17.619631942 +0000 UTC m=+1526.208285601"
Sep 09 20:20:18.660630 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:18.660593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:24.150026 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:24.149985    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:24.162412 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:24.162385    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:24.644285 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:24.644253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:48.607647 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:48.607612    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:20:50.810446 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:50.810406    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:50.829484 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:50.829450    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"]
Sep 09 20:20:50.834913 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:50.834884    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"]
Sep 09 20:20:50.839787 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:20:50.839753    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:20:50.839787 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:20:50.839787 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:20:50.839787 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0" cmd=["/bin/sleep","15"]
Sep 09 20:20:50.840029 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:50.839804    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm" message="" err=<
Sep 09 20:20:50.840029 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:20:50.840029 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:20:50.840029 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:20:50.840029 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:20:50.839823    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:20:50.840029 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:20:50.840029 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:20:50.840029 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm" podUID="da34ca2f-a624-4d1a-a745-4f4cbd4da09d" containerName="main" containerID="cri-o://f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0"
Sep 09 20:20:50.840029 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:50.839869    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm" podUID="da34ca2f-a624-4d1a-a745-4f4cbd4da09d" containerName="main" containerID="cri-o://f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0" gracePeriod=60
Sep 09 20:20:50.905635 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:50.905603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:51.579379 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.579356    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:51.660723 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.660691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-dshm\") pod \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") "
Sep 09 20:20:51.660885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.660735    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tls-certs\") pod \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") "
Sep 09 20:20:51.660885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.660797    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgvhk\" (UniqueName: \"kubernetes.io/projected/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-kube-api-access-rgvhk\") pod \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") "
Sep 09 20:20:51.660885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.660820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-model-cache\") pod \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") "
Sep 09 20:20:51.660885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.660838    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-home\") pod \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") "
Sep 09 20:20:51.660885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.660882    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tmp-dir\") pod \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\" (UID: \"da34ca2f-a624-4d1a-a745-4f4cbd4da09d\") "
Sep 09 20:20:51.661147 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.661122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-model-cache" (OuterVolumeSpecName: "model-cache") pod "da34ca2f-a624-4d1a-a745-4f4cbd4da09d" (UID: "da34ca2f-a624-4d1a-a745-4f4cbd4da09d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:51.661210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.661161    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-home" (OuterVolumeSpecName: "home") pod "da34ca2f-a624-4d1a-a745-4f4cbd4da09d" (UID: "da34ca2f-a624-4d1a-a745-4f4cbd4da09d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:51.661268 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.661226    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da34ca2f-a624-4d1a-a745-4f4cbd4da09d" (UID: "da34ca2f-a624-4d1a-a745-4f4cbd4da09d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:51.663115 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.663094    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-dshm" (OuterVolumeSpecName: "dshm") pod "da34ca2f-a624-4d1a-a745-4f4cbd4da09d" (UID: "da34ca2f-a624-4d1a-a745-4f4cbd4da09d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:20:51.663198 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.663112    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da34ca2f-a624-4d1a-a745-4f4cbd4da09d" (UID: "da34ca2f-a624-4d1a-a745-4f4cbd4da09d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:20:51.663236 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.663221    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-kube-api-access-rgvhk" (OuterVolumeSpecName: "kube-api-access-rgvhk") pod "da34ca2f-a624-4d1a-a745-4f4cbd4da09d" (UID: "da34ca2f-a624-4d1a-a745-4f4cbd4da09d"). InnerVolumeSpecName "kube-api-access-rgvhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:20:51.761612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761525    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:20:51.761612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761556    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:20:51.761612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761569    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:20:51.761612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761581    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgvhk\" (UniqueName: \"kubernetes.io/projected/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-kube-api-access-rgvhk\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:20:51.761612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761592    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:20:51.761612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761604    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da34ca2f-a624-4d1a-a745-4f4cbd4da09d-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:20:51.761612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761603    2578 generic.go:356] "Generic (PLEG): container finished" podID="da34ca2f-a624-4d1a-a745-4f4cbd4da09d" containerID="f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0" exitCode=0
Sep 09 20:20:51.762153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761655    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"
Sep 09 20:20:51.762153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm" event={"ID":"da34ca2f-a624-4d1a-a745-4f4cbd4da09d","Type":"ContainerDied","Data":"f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0"}
Sep 09 20:20:51.762153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm" event={"ID":"da34ca2f-a624-4d1a-a745-4f4cbd4da09d","Type":"ContainerDied","Data":"0bc56cb06ca0be74e96a1a36fc7245313371a126174dca4589069d1afa3de76b"}
Sep 09 20:20:51.762153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.761784    2578 scope.go:117] "RemoveContainer" containerID="f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0"
Sep 09 20:20:51.771507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.771489    2578 scope.go:117] "RemoveContainer" containerID="f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0"
Sep 09 20:20:51.771816 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:20:51.771790    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0\": container with ID starting with f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0 not found: ID does not exist" containerID="f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0"
Sep 09 20:20:51.771920 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.771823    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0"} err="failed to get container status \"f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0\": rpc error: code = NotFound desc = could not find container \"f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0\": container with ID starting with f9d1e82d60a997840104104669aefbe23b2ac8642b7c8686b4c4ec08064fecd0 not found: ID does not exist"
Sep 09 20:20:51.787210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.787184    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"]
Sep 09 20:20:51.791878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.791857    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k59dm"]
Sep 09 20:20:51.896602 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:51.896571    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da34ca2f-a624-4d1a-a745-4f4cbd4da09d" path="/var/lib/kubelet/pods/da34ca2f-a624-4d1a-a745-4f4cbd4da09d/volumes"
Sep 09 20:20:58.608454 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:58.608402    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd" podUID="e9d0e4d2-008a-4833-9771-9a95033170a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:20:58.904871 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:20:58.904789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:01.594224 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.593971    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 09 20:21:01.594224 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:21:01.594000    2578 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd" podUID="e9d0e4d2-008a-4833-9771-9a95033170a6" containerName="main" containerID="cri-o://18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564"
Sep 09 20:21:01.594224 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.594044    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd" podUID="e9d0e4d2-008a-4833-9771-9a95033170a6" containerName="main" containerID="cri-o://18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564" gracePeriod=50
Sep 09 20:21:01.741393 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.741366    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd_e9d0e4d2-008a-4833-9771-9a95033170a6/main/0.log"
Sep 09 20:21:01.743733 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.743716    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:21:01.808057 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.808025    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd_e9d0e4d2-008a-4833-9771-9a95033170a6/main/0.log"
Sep 09 20:21:01.808232 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.808072    2578 generic.go:356] "Generic (PLEG): container finished" podID="e9d0e4d2-008a-4833-9771-9a95033170a6" containerID="18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564" exitCode=1
Sep 09 20:21:01.808232 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.808131    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"
Sep 09 20:21:01.808232 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.808170    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd" event={"ID":"e9d0e4d2-008a-4833-9771-9a95033170a6","Type":"ContainerDied","Data":"18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564"}
Sep 09 20:21:01.808400 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.808253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd" event={"ID":"e9d0e4d2-008a-4833-9771-9a95033170a6","Type":"ContainerDied","Data":"57fcf4859d0148bcb015c41dc031440bc978df3f7df68357a0a73d85d33e7c6f"}
Sep 09 20:21:01.808400 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.808277    2578 scope.go:117] "RemoveContainer" containerID="18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564"
Sep 09 20:21:01.816684 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.816666    2578 scope.go:117] "RemoveContainer" containerID="18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564"
Sep 09 20:21:01.816962 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:21:01.816941    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564\": container with ID starting with 18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564 not found: ID does not exist" containerID="18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564"
Sep 09 20:21:01.817020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.816971    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564"} err="failed to get container status \"18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564\": rpc error: code = NotFound desc = could not find container \"18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564\": container with ID starting with 18e5e1f655a414b991b833a2073ca4d297166c0ef36f988cb2671080959e9564 not found: ID does not exist"
Sep 09 20:21:01.846545 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.846481    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp6nw\" (UniqueName: \"kubernetes.io/projected/e9d0e4d2-008a-4833-9771-9a95033170a6-kube-api-access-tp6nw\") pod \"e9d0e4d2-008a-4833-9771-9a95033170a6\" (UID: \"e9d0e4d2-008a-4833-9771-9a95033170a6\") "
Sep 09 20:21:01.846671 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.846557    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d0e4d2-008a-4833-9771-9a95033170a6-tls-certs\") pod \"e9d0e4d2-008a-4833-9771-9a95033170a6\" (UID: \"e9d0e4d2-008a-4833-9771-9a95033170a6\") "
Sep 09 20:21:01.848531 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.848510    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0e4d2-008a-4833-9771-9a95033170a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9d0e4d2-008a-4833-9771-9a95033170a6" (UID: "e9d0e4d2-008a-4833-9771-9a95033170a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:21:01.848678 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.848659    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d0e4d2-008a-4833-9771-9a95033170a6-kube-api-access-tp6nw" (OuterVolumeSpecName: "kube-api-access-tp6nw") pod "e9d0e4d2-008a-4833-9771-9a95033170a6" (UID: "e9d0e4d2-008a-4833-9771-9a95033170a6"). InnerVolumeSpecName "kube-api-access-tp6nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:01.947506 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.947472    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9d0e4d2-008a-4833-9771-9a95033170a6-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:21:01.947506 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:01.947499    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp6nw\" (UniqueName: \"kubernetes.io/projected/e9d0e4d2-008a-4833-9771-9a95033170a6-kube-api-access-tp6nw\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:21:02.125411 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:02.125331    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"]
Sep 09 20:21:02.131091 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:02.131064    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5vktkd"]
Sep 09 20:21:03.895841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:03.895808    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d0e4d2-008a-4833-9771-9a95033170a6" path="/var/lib/kubelet/pods/e9d0e4d2-008a-4833-9771-9a95033170a6/volumes"
Sep 09 20:21:39.674552 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:39.674518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:41.221431 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:41.221400    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:46.223486 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:46.223399    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:51.941882 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:51.941843    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:52.042904 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:52.042872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:53.650285 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.650251    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"]
Sep 09 20:21:53.650685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.650583    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da34ca2f-a624-4d1a-a745-4f4cbd4da09d" containerName="main"
Sep 09 20:21:53.650685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.650597    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da34ca2f-a624-4d1a-a745-4f4cbd4da09d" containerName="main"
Sep 09 20:21:53.650685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.650616    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0e4d2-008a-4833-9771-9a95033170a6" containerName="main"
Sep 09 20:21:53.650685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.650622    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0e4d2-008a-4833-9771-9a95033170a6" containerName="main"
Sep 09 20:21:53.650821 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.650700    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0e4d2-008a-4833-9771-9a95033170a6" containerName="main"
Sep 09 20:21:53.650821 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.650710    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da34ca2f-a624-4d1a-a745-4f4cbd4da09d" containerName="main"
Sep 09 20:21:53.653512 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.653496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.656159 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.656140    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-lhb4s\""
Sep 09 20:21:53.657174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.657151    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 09 20:21:53.657295 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.657195    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 09 20:21:53.657295 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.657223    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 09 20:21:53.664432 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.664409    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"]
Sep 09 20:21:53.673878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.673857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.673973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.673893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.673973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.673955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.674066 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.674000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.674066 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.674020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.674066 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.674037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2p8\" (UniqueName: \"kubernetes.io/projected/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-kube-api-access-6w2p8\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.676204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.676182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:53.775055 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.775019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.775261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.775059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.775261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.775101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.775261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.775126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.775261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.775147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.775261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.775170    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2p8\" (UniqueName: \"kubernetes.io/projected/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-kube-api-access-6w2p8\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.775502 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.775469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.775550 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.775532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.775646 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.775630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.777468 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.777447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.777928 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.777904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.785380 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.785356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2p8\" (UniqueName: \"kubernetes.io/projected/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-kube-api-access-6w2p8\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-gzwnn\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.960624 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.960541    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"]
Sep 09 20:21:53.963866 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.963848    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:53.964097 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.964076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:21:53.966484 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.966458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-mhjhx\""
Sep 09 20:21:53.977039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.977010    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6tk\" (UniqueName: \"kubernetes.io/projected/00d3a917-39b8-49db-83d1-2a506426efb9-kube-api-access-jq6tk\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:53.977150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.977074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:53.977150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.977117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:53.977150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.977143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d3a917-39b8-49db-83d1-2a506426efb9-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:53.987946 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:53.987917    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"]
Sep 09 20:21:54.018088 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.018058    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:54.078289 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.078252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6tk\" (UniqueName: \"kubernetes.io/projected/00d3a917-39b8-49db-83d1-2a506426efb9-kube-api-access-jq6tk\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:54.078465 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.078301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:54.078465 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.078345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:54.078465 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.078361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d3a917-39b8-49db-83d1-2a506426efb9-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:54.078723 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.078702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:54.078803 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.078780    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:54.080900 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.080883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d3a917-39b8-49db-83d1-2a506426efb9-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:54.091248 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.091219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6tk\" (UniqueName: \"kubernetes.io/projected/00d3a917-39b8-49db-83d1-2a506426efb9-kube-api-access-jq6tk\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:54.255538 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.255454    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"]
Sep 09 20:21:54.259480 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:21:54.259454    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2a9bdb_2497_41b9_b7e1_b0ab0d3ba458.slice/crio-2148c3e2ff55593867ae60d01c6a43284d3f40401a6d074d79b8451be5e5d174 WatchSource:0}: Error finding container 2148c3e2ff55593867ae60d01c6a43284d3f40401a6d074d79b8451be5e5d174: Status 404 returned error can't find the container with id 2148c3e2ff55593867ae60d01c6a43284d3f40401a6d074d79b8451be5e5d174
Sep 09 20:21:54.274428 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.274405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:54.550459 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:54.550434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"]
Sep 09 20:21:54.551985 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:21:54.551961    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d3a917_39b8_49db_83d1_2a506426efb9.slice/crio-98e4ab332e8a1e4c8b64ab180702dad32d49aeb5b26cb615aae17799a968daf3 WatchSource:0}: Error finding container 98e4ab332e8a1e4c8b64ab180702dad32d49aeb5b26cb615aae17799a968daf3: Status 404 returned error can't find the container with id 98e4ab332e8a1e4c8b64ab180702dad32d49aeb5b26cb615aae17799a968daf3
Sep 09 20:21:55.043660 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:55.043619    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" event={"ID":"00d3a917-39b8-49db-83d1-2a506426efb9","Type":"ContainerStarted","Data":"e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8"}
Sep 09 20:21:55.044122 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:55.043682    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" event={"ID":"00d3a917-39b8-49db-83d1-2a506426efb9","Type":"ContainerStarted","Data":"98e4ab332e8a1e4c8b64ab180702dad32d49aeb5b26cb615aae17799a968daf3"}
Sep 09 20:21:55.044122 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:55.043708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:21:55.045261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:55.045240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn" event={"ID":"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458","Type":"ContainerStarted","Data":"0b9431ebdda986a7683759c797bd31f735a08f598b09344404cf4ed29bfcb484"}
Sep 09 20:21:55.045352 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:55.045278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn" event={"ID":"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458","Type":"ContainerStarted","Data":"2148c3e2ff55593867ae60d01c6a43284d3f40401a6d074d79b8451be5e5d174"}
Sep 09 20:21:55.063469 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:55.063412    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" podStartSLOduration=2.063394735 podStartE2EDuration="2.063394735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:55.062433088 +0000 UTC m=+1623.651086767" watchObservedRunningTime="2026-09-09 20:21:55.063394735 +0000 UTC m=+1623.652048391"
Sep 09 20:21:55.099691 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:55.099642    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn" podStartSLOduration=2.099625185 podStartE2EDuration="2.099625185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:55.09898103 +0000 UTC m=+1623.687634686" watchObservedRunningTime="2026-09-09 20:21:55.099625185 +0000 UTC m=+1623.688278842"
Sep 09 20:21:55.964990 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:21:55.964948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:22:03.965058 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:03.965025    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:22:03.977777 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:03.977747    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:22:04.096609 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:04.096582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:22:04.385636 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:04.385594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:05.740205 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:05.740169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:05.813101 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:05.813071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:26.051484 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:26.051449    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:22:27.924616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:27.924582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:27.934101 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:27.934069    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"]
Sep 09 20:22:27.938130 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:27.938105    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"]
Sep 09 20:22:27.943599 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:22:27.943566    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:22:27.943599 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:22:27.943599 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:22:27.943599 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="0b9431ebdda986a7683759c797bd31f735a08f598b09344404cf4ed29bfcb484" cmd=["/bin/sleep","15"]
Sep 09 20:22:27.943828 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:27.943619    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn" message="" err=<
Sep 09 20:22:27.943828 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:22:27.943828 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:22:27.943828 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:22:27.943828 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:22:27.943636    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:22:27.943828 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:22:27.943828 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:22:27.943828 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn" podUID="9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" containerName="main" containerID="cri-o://0b9431ebdda986a7683759c797bd31f735a08f598b09344404cf4ed29bfcb484"
Sep 09 20:22:27.943828 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:27.943677    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn" podUID="9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" containerName="main" containerID="cri-o://0b9431ebdda986a7683759c797bd31f735a08f598b09344404cf4ed29bfcb484" gracePeriod=60
Sep 09 20:22:28.021054 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.021025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:28.086174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.086135    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn" podUID="9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 09 20:22:28.189625 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.189591    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" containerID="0b9431ebdda986a7683759c797bd31f735a08f598b09344404cf4ed29bfcb484" exitCode=0
Sep 09 20:22:28.189741 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.189669    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn" event={"ID":"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458","Type":"ContainerDied","Data":"0b9431ebdda986a7683759c797bd31f735a08f598b09344404cf4ed29bfcb484"}
Sep 09 20:22:28.189741 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.189721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn" event={"ID":"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458","Type":"ContainerDied","Data":"2148c3e2ff55593867ae60d01c6a43284d3f40401a6d074d79b8451be5e5d174"}
Sep 09 20:22:28.189741 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.189740    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2148c3e2ff55593867ae60d01c6a43284d3f40401a6d074d79b8451be5e5d174"
Sep 09 20:22:28.196210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.196190    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:22:28.266040 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266008    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-model-cache\") pod \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") "
Sep 09 20:22:28.266233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266050    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tmp-dir\") pod \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") "
Sep 09 20:22:28.266233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266084    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tls-certs\") pod \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") "
Sep 09 20:22:28.266233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-home\") pod \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") "
Sep 09 20:22:28.266233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-dshm\") pod \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") "
Sep 09 20:22:28.266233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266220    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w2p8\" (UniqueName: \"kubernetes.io/projected/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-kube-api-access-6w2p8\") pod \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\" (UID: \"9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458\") "
Sep 09 20:22:28.266527 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266283    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" (UID: "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:28.266527 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266334    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" (UID: "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:28.266527 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266453    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-home" (OuterVolumeSpecName: "home") pod "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" (UID: "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:28.266527 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266506    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:22:28.266527 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.266524    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:22:28.268878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.268827    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-kube-api-access-6w2p8" (OuterVolumeSpecName: "kube-api-access-6w2p8") pod "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" (UID: "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458"). InnerVolumeSpecName "kube-api-access-6w2p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:28.268999 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.268919    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" (UID: "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:28.268999 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.268920    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-dshm" (OuterVolumeSpecName: "dshm") pod "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" (UID: "9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:28.367749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.367709    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:22:28.367749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.367741    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:22:28.367749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.367751    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2p8\" (UniqueName: \"kubernetes.io/projected/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-kube-api-access-6w2p8\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:22:28.367993 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:28.367762    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:22:29.195095 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:29.195062    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"
Sep 09 20:22:29.219383 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:29.219349    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"]
Sep 09 20:22:29.226129 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:29.226102    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-gzwnn"]
Sep 09 20:22:29.896814 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:29.896775    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" path="/var/lib/kubelet/pods/9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458/volumes"
Sep 09 20:22:36.051612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:36.051567    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" podUID="00d3a917-39b8-49db-83d1-2a506426efb9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:22:42.957658 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:42.957591    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" podUID="00d3a917-39b8-49db-83d1-2a506426efb9" containerName="main" containerID="cri-o://e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8" gracePeriod=45
Sep 09 20:22:46.051151 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:46.051105    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" podUID="00d3a917-39b8-49db-83d1-2a506426efb9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:22:52.361241 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:52.361210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:52.464182 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:52.464142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:52.599655 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:52.599603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:54.909293 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:54.909235    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:22:56.050721 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:56.050680    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" podUID="00d3a917-39b8-49db-83d1-2a506426efb9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:22:56.051237 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:56.050826    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:22:59.069839 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:22:59.069807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:05.237362 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:05.237330    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:05.321147 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:05.321115    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:06.051509 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:06.051470    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" podUID="00d3a917-39b8-49db-83d1-2a506426efb9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:23:08.506264 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:08.506232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:13.210241 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.210216    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:23:13.363778 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.363753    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq6tk\" (UniqueName: \"kubernetes.io/projected/00d3a917-39b8-49db-83d1-2a506426efb9-kube-api-access-jq6tk\") pod \"00d3a917-39b8-49db-83d1-2a506426efb9\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") "
Sep 09 20:23:13.363970 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.363822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-tokenizer-cache\") pod \"00d3a917-39b8-49db-83d1-2a506426efb9\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") "
Sep 09 20:23:13.363970 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.363839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d3a917-39b8-49db-83d1-2a506426efb9-tls-certs\") pod \"00d3a917-39b8-49db-83d1-2a506426efb9\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") "
Sep 09 20:23:13.363970 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.363886    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-cachi2-cache\") pod \"00d3a917-39b8-49db-83d1-2a506426efb9\" (UID: \"00d3a917-39b8-49db-83d1-2a506426efb9\") "
Sep 09 20:23:13.364142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.364090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "00d3a917-39b8-49db-83d1-2a506426efb9" (UID: "00d3a917-39b8-49db-83d1-2a506426efb9"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:23:13.364204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.364180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "00d3a917-39b8-49db-83d1-2a506426efb9" (UID: "00d3a917-39b8-49db-83d1-2a506426efb9"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:23:13.365950 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.365919    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d3a917-39b8-49db-83d1-2a506426efb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00d3a917-39b8-49db-83d1-2a506426efb9" (UID: "00d3a917-39b8-49db-83d1-2a506426efb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:13.366059 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.365980    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d3a917-39b8-49db-83d1-2a506426efb9-kube-api-access-jq6tk" (OuterVolumeSpecName: "kube-api-access-jq6tk") pod "00d3a917-39b8-49db-83d1-2a506426efb9" (UID: "00d3a917-39b8-49db-83d1-2a506426efb9"). InnerVolumeSpecName "kube-api-access-jq6tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:13.385565 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.385542    2578 generic.go:356] "Generic (PLEG): container finished" podID="00d3a917-39b8-49db-83d1-2a506426efb9" containerID="e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8" exitCode=0
Sep 09 20:23:13.385672 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.385579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" event={"ID":"00d3a917-39b8-49db-83d1-2a506426efb9","Type":"ContainerDied","Data":"e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8"}
Sep 09 20:23:13.385672 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.385585    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"
Sep 09 20:23:13.385672 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.385620    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf" event={"ID":"00d3a917-39b8-49db-83d1-2a506426efb9","Type":"ContainerDied","Data":"98e4ab332e8a1e4c8b64ab180702dad32d49aeb5b26cb615aae17799a968daf3"}
Sep 09 20:23:13.385672 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.385643    2578 scope.go:117] "RemoveContainer" containerID="e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8"
Sep 09 20:23:13.393870 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.393845    2578 scope.go:117] "RemoveContainer" containerID="e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8"
Sep 09 20:23:13.394178 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:23:13.394151    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8\": container with ID starting with e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8 not found: ID does not exist" containerID="e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8"
Sep 09 20:23:13.394260 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.394184    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8"} err="failed to get container status \"e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8\": rpc error: code = NotFound desc = could not find container \"e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8\": container with ID starting with e3a3e1f893551ff6781f722772c4ab8b69cf2b2cf254b3b4891386ad002bdaa8 not found: ID does not exist"
Sep 09 20:23:13.408266 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.408246    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"]
Sep 09 20:23:13.413009 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.412989    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcrkcbf"]
Sep 09 20:23:13.465211 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.465185    2578 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-tokenizer-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:23:13.465211 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.465213    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d3a917-39b8-49db-83d1-2a506426efb9-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:23:13.465391 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.465224    2578 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d3a917-39b8-49db-83d1-2a506426efb9-cachi2-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:23:13.465391 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.465234    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq6tk\" (UniqueName: \"kubernetes.io/projected/00d3a917-39b8-49db-83d1-2a506426efb9-kube-api-access-jq6tk\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:23:13.896246 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:13.896212    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d3a917-39b8-49db-83d1-2a506426efb9" path="/var/lib/kubelet/pods/00d3a917-39b8-49db-83d1-2a506426efb9/volumes"
Sep 09 20:23:17.952558 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:17.952525    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:18.041732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:18.041703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:24.101845 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:24.101812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:29.947816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:29.947779    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:30.018532 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:30.018490    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:33.877750 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.877716    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"]
Sep 09 20:23:33.878145 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.878042    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d3a917-39b8-49db-83d1-2a506426efb9" containerName="main"
Sep 09 20:23:33.878145 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.878054    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d3a917-39b8-49db-83d1-2a506426efb9" containerName="main"
Sep 09 20:23:33.878145 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.878071    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" containerName="main"
Sep 09 20:23:33.878145 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.878076    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" containerName="main"
Sep 09 20:23:33.878145 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.878138    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d3a917-39b8-49db-83d1-2a506426efb9" containerName="main"
Sep 09 20:23:33.878145 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.878147    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d2a9bdb-2497-41b9-b7e1-b0ab0d3ba458" containerName="main"
Sep 09 20:23:33.880978 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.880958    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:33.884603 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.884569    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-xf6w4\""
Sep 09 20:23:33.884603 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.884590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 09 20:23:33.884813 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.884633    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 09 20:23:33.884813 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.884673    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-q2x8k\""
Sep 09 20:23:33.884813 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.884568    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 09 20:23:33.896916 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.896892    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"]
Sep 09 20:23:33.933568 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.933533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:33.933568 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.933572    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:33.933759 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.933601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-home\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:33.933759 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.933646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26wvt\" (UniqueName: \"kubernetes.io/projected/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-kube-api-access-26wvt\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:33.933759 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.933670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:33.933759 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:33.933713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.034453 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.034416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.034453 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.034461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.034742 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.034492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-home\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.034742 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.034514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26wvt\" (UniqueName: \"kubernetes.io/projected/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-kube-api-access-26wvt\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.034742 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.034540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.034742 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.034569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.034960 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.034919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-home\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.035020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.034979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.035071 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.035012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.036152 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.036130    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:34.037155 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.037130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.037443 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.037425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.045651 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.045623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26wvt\" (UniqueName: \"kubernetes.io/projected/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-kube-api-access-26wvt\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.191004 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.190903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:34.255080 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.255048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:34.467020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.466994    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"]
Sep 09 20:23:34.469376 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:23:34.469350    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a1cd89_422f_41b1_bd95_7ffdc1efcbaf.slice/crio-bc2d5a838fc36e7cad8170ac2f90d676c6cc16774396e1531b0484936045f409 WatchSource:0}: Error finding container bc2d5a838fc36e7cad8170ac2f90d676c6cc16774396e1531b0484936045f409: Status 404 returned error can't find the container with id bc2d5a838fc36e7cad8170ac2f90d676c6cc16774396e1531b0484936045f409
Sep 09 20:23:34.482958 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:34.482934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" event={"ID":"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf","Type":"ContainerStarted","Data":"bc2d5a838fc36e7cad8170ac2f90d676c6cc16774396e1531b0484936045f409"}
Sep 09 20:23:36.497286 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:36.497191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" event={"ID":"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf","Type":"ContainerStarted","Data":"4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7"}
Sep 09 20:23:36.497756 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:36.497373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:37.503644 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:37.503608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" event={"ID":"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf","Type":"ContainerStarted","Data":"df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd"}
Sep 09 20:23:37.523690 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:37.523640    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" podStartSLOduration=2.788543233 podStartE2EDuration="4.523623029s" totalImagesPullingTime="1.735079796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:37.523576467 +0000 UTC m=+1726.112230120" watchObservedRunningTime="2026-09-09 20:23:37.523623029 +0000 UTC m=+1726.112276689"
Sep 09 20:23:38.191471 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:38.191435    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:44.191300 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:44.191253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:44.203594 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:44.203567    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:44.544186 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:44.544105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:23:48.616284 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:48.616249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:55.549233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:23:55.549199    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:24:02.877827 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:02.877790    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:47.291457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:47.291366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:47.416621 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:47.416582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:47.543556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:47.543452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:47.767787 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:47.767746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:48.087862 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:48.087825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:51.810564 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.810534    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"]
Sep 09 20:24:51.813948 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.813931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:51.819023 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.819002    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 09 20:24:51.819842 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.819824    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 09 20:24:51.819994 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.819974    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-454ck\""
Sep 09 20:24:51.821982 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.821967    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 09 20:24:51.834905 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.834883    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"]
Sep 09 20:24:51.901444 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.901417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:51.904609 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.904587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-dshm\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:51.904737 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.904619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzqv\" (UniqueName: \"kubernetes.io/projected/448d386c-9365-4c97-9182-50d1806153cd-kube-api-access-nvzqv\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:51.904737 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.904714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448d386c-9365-4c97-9182-50d1806153cd-tls-certs\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:51.904855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.904766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-model-cache\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:51.904855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.904803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-home\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:51.904855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.904835    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:51.964972 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.964939    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:24:51.967111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:51.967086    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:24:52.005661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.005627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-dshm\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.005829 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.005665    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzqv\" (UniqueName: \"kubernetes.io/projected/448d386c-9365-4c97-9182-50d1806153cd-kube-api-access-nvzqv\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.005829 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.005734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448d386c-9365-4c97-9182-50d1806153cd-tls-certs\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.005829 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.005781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-model-cache\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.005829 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.005815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-home\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.006011 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.005844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.006180 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.006155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-model-cache\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.006261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.006174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-home\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.006261 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.006238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.007891 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.007870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-dshm\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.008292 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.008273    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448d386c-9365-4c97-9182-50d1806153cd-tls-certs\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.015013 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.014987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzqv\" (UniqueName: \"kubernetes.io/projected/448d386c-9365-4c97-9182-50d1806153cd-kube-api-access-nvzqv\") pod \"test-div-v1-kserve-fb566d667-228wj\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.124148 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.124116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:24:52.402010 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.401979    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"]
Sep 09 20:24:52.405077 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:24:52.405046    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448d386c_9365_4c97_9182_50d1806153cd.slice/crio-e6bec0b462cb827b2420a70a6e24fd67c81d5e09e1aeefaad5017c0bf48e8a0a WatchSource:0}: Error finding container e6bec0b462cb827b2420a70a6e24fd67c81d5e09e1aeefaad5017c0bf48e8a0a: Status 404 returned error can't find the container with id e6bec0b462cb827b2420a70a6e24fd67c81d5e09e1aeefaad5017c0bf48e8a0a
Sep 09 20:24:52.614871 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.614841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:52.844703 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:52.844653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj" event={"ID":"448d386c-9365-4c97-9182-50d1806153cd","Type":"ContainerStarted","Data":"e6bec0b462cb827b2420a70a6e24fd67c81d5e09e1aeefaad5017c0bf48e8a0a"}
Sep 09 20:24:53.851510 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:53.851480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj" event={"ID":"448d386c-9365-4c97-9182-50d1806153cd","Type":"ContainerStarted","Data":"31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24"}
Sep 09 20:24:53.872926 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:53.872876    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj" podStartSLOduration=1.8385616489999999 podStartE2EDuration="2.872860893s" totalImagesPullingTime="1.034299244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:53.872283925 +0000 UTC m=+1802.460937582" watchObservedRunningTime="2026-09-09 20:24:53.872860893 +0000 UTC m=+1802.461514546"
Sep 09 20:24:54.124751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:24:54.124659    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:25:01.028869 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:01.028836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:02.124860 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:02.124824    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:25:02.137169 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:02.137145    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:25:02.903251 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:02.903218    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:25:04.749124 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:04.749091    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:20.389832 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.389800    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:20.398030 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.398000    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"]
Sep 09 20:25:20.407910 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:25:20.407864    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:25:20.407910 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:25:20.407910 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:25:20.407910 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd" cmd=["/bin/sleep","15"]
Sep 09 20:25:20.408164 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.407929    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" message="" err=<
Sep 09 20:25:20.408164 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:25:20.408164 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:25:20.408164 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:25:20.408164 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:25:20.407941    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:25:20.408164 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:25:20.408164 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:25:20.408164 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="main" containerID="cri-o://df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd"
Sep 09 20:25:20.408164 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.407983    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="main" containerID="cri-o://df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd" gracePeriod=60
Sep 09 20:25:20.476184 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.476149    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:20.533709 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.533664    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8001/health\": dial tcp 10.132.0.42:8001: connect: connection refused"
Sep 09 20:25:20.625491 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.625431    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="llm-d-routing-sidecar" containerID="cri-o://4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7" gracePeriod=60
Sep 09 20:25:20.877230 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.877199    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:25:20.963973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.963940    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tls-certs\") pod \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") "
Sep 09 20:25:20.963973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.963980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-home\") pod \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") "
Sep 09 20:25:20.964221 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.964047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-model-cache\") pod \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") "
Sep 09 20:25:20.964221 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.964064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tmp-dir\") pod \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") "
Sep 09 20:25:20.964221 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.964087    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26wvt\" (UniqueName: \"kubernetes.io/projected/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-kube-api-access-26wvt\") pod \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") "
Sep 09 20:25:20.964221 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.964115    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-dshm\") pod \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\" (UID: \"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf\") "
Sep 09 20:25:20.964814 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.964384    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" (UID: "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:20.964814 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.964400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-home" (OuterVolumeSpecName: "home") pod "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" (UID: "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:20.964814 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.964406    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" (UID: "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:20.966301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.966278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" (UID: "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:20.966718 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.966697    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-dshm" (OuterVolumeSpecName: "dshm") pod "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" (UID: "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:20.966791 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.966726    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-kube-api-access-26wvt" (OuterVolumeSpecName: "kube-api-access-26wvt") pod "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" (UID: "59a1cd89-422f-41b1-bd95-7ffdc1efcbaf"). InnerVolumeSpecName "kube-api-access-26wvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:20.973950 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.973925    2578 generic.go:356] "Generic (PLEG): container finished" podID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerID="df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd" exitCode=0
Sep 09 20:25:20.973950 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.973948    2578 generic.go:356] "Generic (PLEG): container finished" podID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerID="4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7" exitCode=0
Sep 09 20:25:20.974086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.973974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"
Sep 09 20:25:20.974086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.974008    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" event={"ID":"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf","Type":"ContainerDied","Data":"df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd"}
Sep 09 20:25:20.974086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.974065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" event={"ID":"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf","Type":"ContainerDied","Data":"4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7"}
Sep 09 20:25:20.974086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.974081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2" event={"ID":"59a1cd89-422f-41b1-bd95-7ffdc1efcbaf","Type":"ContainerDied","Data":"bc2d5a838fc36e7cad8170ac2f90d676c6cc16774396e1531b0484936045f409"}
Sep 09 20:25:20.974247 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.974101    2578 scope.go:117] "RemoveContainer" containerID="df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd"
Sep 09 20:25:20.983549 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.983533    2578 scope.go:117] "RemoveContainer" containerID="4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7"
Sep 09 20:25:20.992295 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.992277    2578 scope.go:117] "RemoveContainer" containerID="df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd"
Sep 09 20:25:20.992594 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:25:20.992575    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd\": container with ID starting with df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd not found: ID does not exist" containerID="df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd"
Sep 09 20:25:20.992657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.992603    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd"} err="failed to get container status \"df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd\": rpc error: code = NotFound desc = could not find container \"df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd\": container with ID starting with df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd not found: ID does not exist"
Sep 09 20:25:20.992657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.992619    2578 scope.go:117] "RemoveContainer" containerID="4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7"
Sep 09 20:25:20.992899 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:25:20.992883    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7\": container with ID starting with 4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7 not found: ID does not exist" containerID="4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7"
Sep 09 20:25:20.992936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.992903    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7"} err="failed to get container status \"4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7\": rpc error: code = NotFound desc = could not find container \"4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7\": container with ID starting with 4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7 not found: ID does not exist"
Sep 09 20:25:20.992936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.992916    2578 scope.go:117] "RemoveContainer" containerID="df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd"
Sep 09 20:25:20.993102 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.993086    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd"} err="failed to get container status \"df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd\": rpc error: code = NotFound desc = could not find container \"df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd\": container with ID starting with df512178c0482f5705beee8cfc4d9bca946c725a92b9b535b55d6113eaa017cd not found: ID does not exist"
Sep 09 20:25:20.993149 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.993103    2578 scope.go:117] "RemoveContainer" containerID="4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7"
Sep 09 20:25:20.993325 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:20.993287    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7"} err="failed to get container status \"4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7\": rpc error: code = NotFound desc = could not find container \"4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7\": container with ID starting with 4d280302db3784ca99259272cba094bb45406a44fbc2529eef57f843f85022f7 not found: ID does not exist"
Sep 09 20:25:21.001286 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:21.001262    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"]
Sep 09 20:25:21.006331 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:21.006295    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-w74c2"]
Sep 09 20:25:21.065130 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:21.065101    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:21.065130 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:21.065132    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:21.065318 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:21.065145    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26wvt\" (UniqueName: \"kubernetes.io/projected/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-kube-api-access-26wvt\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:21.065318 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:21.065158    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:21.065318 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:21.065174    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:21.065318 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:21.065187    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:21.896770 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:21.896736    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" path="/var/lib/kubelet/pods/59a1cd89-422f-41b1-bd95-7ffdc1efcbaf/volumes"
Sep 09 20:25:23.090479 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:23.090441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:23.125530 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:23.125499    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"]
Sep 09 20:25:23.197549 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:23.197507    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:23.288449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:23.288413    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:23.600264 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:23.600228    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:28.259949 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:28.259907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:29.822600 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:29.822562    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:31.939342 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.939293    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"]
Sep 09 20:25:31.939829 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.939744    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="llm-d-routing-sidecar"
Sep 09 20:25:31.939829 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.939763    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="llm-d-routing-sidecar"
Sep 09 20:25:31.939829 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.939785    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="main"
Sep 09 20:25:31.939829 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.939793    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="main"
Sep 09 20:25:31.940043 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.939885    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="llm-d-routing-sidecar"
Sep 09 20:25:31.940043 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.939897    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="59a1cd89-422f-41b1-bd95-7ffdc1efcbaf" containerName="main"
Sep 09 20:25:31.943927 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.943906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:31.946497 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.946477    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 09 20:25:31.947470 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.947450    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 09 20:25:31.947545 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.947481    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 09 20:25:31.947545 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.947538    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-q7hjf\""
Sep 09 20:25:31.951800 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:31.951781    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"]
Sep 09 20:25:32.050791 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.050760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:32.053898 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.053875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-home\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.054014 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.053911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.054014 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.053953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-dshm\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.054014 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.053975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-model-cache\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.054014 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.053997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd77d16-0421-47dd-92d0-cace3f795a45-tls-certs\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.054175 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.054134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhg24\" (UniqueName: \"kubernetes.io/projected/5cd77d16-0421-47dd-92d0-cace3f795a45-kube-api-access-bhg24\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.154824 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.154785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhg24\" (UniqueName: \"kubernetes.io/projected/5cd77d16-0421-47dd-92d0-cace3f795a45-kube-api-access-bhg24\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.155022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.154846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-home\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.155022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.154874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.155022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.154907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-dshm\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.155022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.154921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-model-cache\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.155022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.154940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd77d16-0421-47dd-92d0-cace3f795a45-tls-certs\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.155353 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.155303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-home\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.155431 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.155408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-model-cache\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.155472 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.155438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.157414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.157389    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-dshm\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.157624 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.157606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd77d16-0421-47dd-92d0-cace3f795a45-tls-certs\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.163676 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.163647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhg24\" (UniqueName: \"kubernetes.io/projected/5cd77d16-0421-47dd-92d0-cace3f795a45-kube-api-access-bhg24\") pod \"test-div-v1-kserve-fb566d667-zkfrw\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.255300 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.255196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:32.550868 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:32.550782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"]
Sep 09 20:25:32.555570 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:25:32.555539    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd77d16_0421_47dd_92d0_cace3f795a45.slice/crio-01bec6c1e2b881a769d56ac2dc3fa063d05d7da17042d1808d1a6a3977c9e472 WatchSource:0}: Error finding container 01bec6c1e2b881a769d56ac2dc3fa063d05d7da17042d1808d1a6a3977c9e472: Status 404 returned error can't find the container with id 01bec6c1e2b881a769d56ac2dc3fa063d05d7da17042d1808d1a6a3977c9e472
Sep 09 20:25:33.028896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:33.028853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw" event={"ID":"5cd77d16-0421-47dd-92d0-cace3f795a45","Type":"ContainerStarted","Data":"356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace"}
Sep 09 20:25:33.029283 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:33.028957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw" event={"ID":"5cd77d16-0421-47dd-92d0-cace3f795a45","Type":"ContainerStarted","Data":"01bec6c1e2b881a769d56ac2dc3fa063d05d7da17042d1808d1a6a3977c9e472"}
Sep 09 20:25:33.049716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:33.049658    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw" podStartSLOduration=2.04964027 podStartE2EDuration="2.04964027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:33.048207932 +0000 UTC m=+1841.636861588" watchObservedRunningTime="2026-09-09 20:25:33.04964027 +0000 UTC m=+1841.638293926"
Sep 09 20:25:33.255281 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:33.255249    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:35.702015 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:35.701983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:35.802730 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:35.802691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:38.145661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.145624    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj" podUID="448d386c-9365-4c97-9182-50d1806153cd" containerName="main" containerID="cri-o://31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24" gracePeriod=45
Sep 09 20:25:38.387690 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.387660    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:25:38.511987 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.511889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448d386c-9365-4c97-9182-50d1806153cd-tls-certs\") pod \"448d386c-9365-4c97-9182-50d1806153cd\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") "
Sep 09 20:25:38.511987 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.511941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-dshm\") pod \"448d386c-9365-4c97-9182-50d1806153cd\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") "
Sep 09 20:25:38.511987 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.511987    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvzqv\" (UniqueName: \"kubernetes.io/projected/448d386c-9365-4c97-9182-50d1806153cd-kube-api-access-nvzqv\") pod \"448d386c-9365-4c97-9182-50d1806153cd\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") "
Sep 09 20:25:38.512276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.512003    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-model-cache\") pod \"448d386c-9365-4c97-9182-50d1806153cd\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") "
Sep 09 20:25:38.512276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.512019    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-tmp-dir\") pod \"448d386c-9365-4c97-9182-50d1806153cd\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") "
Sep 09 20:25:38.512276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.512045    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-home\") pod \"448d386c-9365-4c97-9182-50d1806153cd\" (UID: \"448d386c-9365-4c97-9182-50d1806153cd\") "
Sep 09 20:25:38.512276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.512255    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "448d386c-9365-4c97-9182-50d1806153cd" (UID: "448d386c-9365-4c97-9182-50d1806153cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:38.512487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.512290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "448d386c-9365-4c97-9182-50d1806153cd" (UID: "448d386c-9365-4c97-9182-50d1806153cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:38.512487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.512386    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-home" (OuterVolumeSpecName: "home") pod "448d386c-9365-4c97-9182-50d1806153cd" (UID: "448d386c-9365-4c97-9182-50d1806153cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:38.514234 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.514204    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-dshm" (OuterVolumeSpecName: "dshm") pod "448d386c-9365-4c97-9182-50d1806153cd" (UID: "448d386c-9365-4c97-9182-50d1806153cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:38.514234 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.514226    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448d386c-9365-4c97-9182-50d1806153cd-kube-api-access-nvzqv" (OuterVolumeSpecName: "kube-api-access-nvzqv") pod "448d386c-9365-4c97-9182-50d1806153cd" (UID: "448d386c-9365-4c97-9182-50d1806153cd"). InnerVolumeSpecName "kube-api-access-nvzqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:38.514420 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.514204    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448d386c-9365-4c97-9182-50d1806153cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "448d386c-9365-4c97-9182-50d1806153cd" (UID: "448d386c-9365-4c97-9182-50d1806153cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:38.612765 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.612718    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448d386c-9365-4c97-9182-50d1806153cd-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:38.612765 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.612757    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:38.612765 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.612767    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvzqv\" (UniqueName: \"kubernetes.io/projected/448d386c-9365-4c97-9182-50d1806153cd-kube-api-access-nvzqv\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:38.612765 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.612777    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:38.613032 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.612786    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:38.613032 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:38.612793    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448d386c-9365-4c97-9182-50d1806153cd-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:25:39.058949 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.058917    2578 generic.go:356] "Generic (PLEG): container finished" podID="448d386c-9365-4c97-9182-50d1806153cd" containerID="31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24" exitCode=0
Sep 09 20:25:39.059190 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.058981    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"
Sep 09 20:25:39.059190 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.059005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj" event={"ID":"448d386c-9365-4c97-9182-50d1806153cd","Type":"ContainerDied","Data":"31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24"}
Sep 09 20:25:39.059190 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.059079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj" event={"ID":"448d386c-9365-4c97-9182-50d1806153cd","Type":"ContainerDied","Data":"e6bec0b462cb827b2420a70a6e24fd67c81d5e09e1aeefaad5017c0bf48e8a0a"}
Sep 09 20:25:39.059190 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.059116    2578 scope.go:117] "RemoveContainer" containerID="31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24"
Sep 09 20:25:39.071161 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.071132    2578 scope.go:117] "RemoveContainer" containerID="31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24"
Sep 09 20:25:39.071508 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:25:39.071463    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24\": container with ID starting with 31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24 not found: ID does not exist" containerID="31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24"
Sep 09 20:25:39.071635 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.071507    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24"} err="failed to get container status \"31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24\": rpc error: code = NotFound desc = could not find container \"31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24\": container with ID starting with 31ecf77da2a42baf2add363c1fba014131bb71667691f5bfb4ce3264e176ed24 not found: ID does not exist"
Sep 09 20:25:39.086860 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.086838    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"]
Sep 09 20:25:39.093902 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.093882    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-228wj"]
Sep 09 20:25:39.896072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:39.896038    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="448d386c-9365-4c97-9182-50d1806153cd" path="/var/lib/kubelet/pods/448d386c-9365-4c97-9182-50d1806153cd/volumes"
Sep 09 20:25:40.223260 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.223166    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"]
Sep 09 20:25:40.223603 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.223584    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="448d386c-9365-4c97-9182-50d1806153cd" containerName="main"
Sep 09 20:25:40.223691 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.223607    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="448d386c-9365-4c97-9182-50d1806153cd" containerName="main"
Sep 09 20:25:40.223691 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.223685    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="448d386c-9365-4c97-9182-50d1806153cd" containerName="main"
Sep 09 20:25:40.227812 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.227788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.230430 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.230407    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 09 20:25:40.238353 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.238331    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"]
Sep 09 20:25:40.255896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.255872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:40.327616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.327584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.327789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.327622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5934576-2058-4ea7-bf91-c8e62d7fc379-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.327789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.327717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.327789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.327738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.327789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.327768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-home\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.327928 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.327805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljljb\" (UniqueName: \"kubernetes.io/projected/b5934576-2058-4ea7-bf91-c8e62d7fc379-kube-api-access-ljljb\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.428518 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.428484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.428688 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.428523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.428688 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.428543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-home\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.428688 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.428591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljljb\" (UniqueName: \"kubernetes.io/projected/b5934576-2058-4ea7-bf91-c8e62d7fc379-kube-api-access-ljljb\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.428688 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.428639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.428688 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.428657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5934576-2058-4ea7-bf91-c8e62d7fc379-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.429006 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.428973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.429075 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.429025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-home\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.429075 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.429048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.430846 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.430817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.431102 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.431087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5934576-2058-4ea7-bf91-c8e62d7fc379-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.445723 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.445701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljljb\" (UniqueName: \"kubernetes.io/projected/b5934576-2058-4ea7-bf91-c8e62d7fc379-kube-api-access-ljljb\") pod \"test-div-v2-kserve-5f6fdb6b77-pqc77\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.539458 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.539371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:40.812602 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:40.812569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"]
Sep 09 20:25:40.814391 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:25:40.814362    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5934576_2058_4ea7_bf91_c8e62d7fc379.slice/crio-6a8f32f9462d64247156387e7090636012bbf28a507ca6b518a94c786ff44fc0 WatchSource:0}: Error finding container 6a8f32f9462d64247156387e7090636012bbf28a507ca6b518a94c786ff44fc0: Status 404 returned error can't find the container with id 6a8f32f9462d64247156387e7090636012bbf28a507ca6b518a94c786ff44fc0
Sep 09 20:25:41.071639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:41.071551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77" event={"ID":"b5934576-2058-4ea7-bf91-c8e62d7fc379","Type":"ContainerStarted","Data":"07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89"}
Sep 09 20:25:41.071639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:41.071623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77" event={"ID":"b5934576-2058-4ea7-bf91-c8e62d7fc379","Type":"ContainerStarted","Data":"6a8f32f9462d64247156387e7090636012bbf28a507ca6b518a94c786ff44fc0"}
Sep 09 20:25:41.098340 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:41.098261    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77" podStartSLOduration=1.098239502 podStartE2EDuration="1.098239502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:41.09560932 +0000 UTC m=+1849.684262979" watchObservedRunningTime="2026-09-09 20:25:41.098239502 +0000 UTC m=+1849.686893160"
Sep 09 20:25:41.539591 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:41.539559    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:42.256092 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:42.256042    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:42.268538 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:42.268512    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:43.091619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:43.091589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:25:50.540205 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:50.540159    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:50.552783 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:50.552758    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:51.130908 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:51.130880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:25:57.942498 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:57.942463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:58.064777 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:58.064735    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:58.383600 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:58.383564    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:58.495858 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:58.495822    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:58.591391 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:58.591358    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"]
Sep 09 20:25:58.597054 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:58.597023    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:58.696887 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:58.696807    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:58.836158 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:58.836102    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"]
Sep 09 20:25:58.853499 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:25:58.853356    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:02.933196 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:02.933163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:04.046925 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:04.046893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:05.084039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:05.084010    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:09.750280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:09.750237    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:09.847206 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:09.847164    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:09.977668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:09.977616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:10.089085 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:10.088994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:10.921251 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:10.921220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:11.019839 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:11.019811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:13.408202 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.408165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:13.509783 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.509751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:13.612256 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.612224    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77" podUID="b5934576-2058-4ea7-bf91-c8e62d7fc379" containerName="main" containerID="cri-o://07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89" gracePeriod=45
Sep 09 20:26:13.850184 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.850159    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:26:13.875102 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.875071    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw" podUID="5cd77d16-0421-47dd-92d0-cace3f795a45" containerName="main" containerID="cri-o://356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace" gracePeriod=45
Sep 09 20:26:13.904816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.904793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljljb\" (UniqueName: \"kubernetes.io/projected/b5934576-2058-4ea7-bf91-c8e62d7fc379-kube-api-access-ljljb\") pod \"b5934576-2058-4ea7-bf91-c8e62d7fc379\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") "
Sep 09 20:26:13.904931 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.904880    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-dshm\") pod \"b5934576-2058-4ea7-bf91-c8e62d7fc379\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") "
Sep 09 20:26:13.904931 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.904904    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-tmp-dir\") pod \"b5934576-2058-4ea7-bf91-c8e62d7fc379\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") "
Sep 09 20:26:13.904931 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.904921    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5934576-2058-4ea7-bf91-c8e62d7fc379-tls-certs\") pod \"b5934576-2058-4ea7-bf91-c8e62d7fc379\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") "
Sep 09 20:26:13.905081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.904943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-home\") pod \"b5934576-2058-4ea7-bf91-c8e62d7fc379\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") "
Sep 09 20:26:13.905081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.904988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-model-cache\") pod \"b5934576-2058-4ea7-bf91-c8e62d7fc379\" (UID: \"b5934576-2058-4ea7-bf91-c8e62d7fc379\") "
Sep 09 20:26:13.905210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.905186    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5934576-2058-4ea7-bf91-c8e62d7fc379" (UID: "b5934576-2058-4ea7-bf91-c8e62d7fc379"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:13.905571 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.905296    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-home" (OuterVolumeSpecName: "home") pod "b5934576-2058-4ea7-bf91-c8e62d7fc379" (UID: "b5934576-2058-4ea7-bf91-c8e62d7fc379"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:13.905571 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.905344    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:13.905720 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.905615    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5934576-2058-4ea7-bf91-c8e62d7fc379" (UID: "b5934576-2058-4ea7-bf91-c8e62d7fc379"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:13.907123 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.907104    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5934576-2058-4ea7-bf91-c8e62d7fc379-kube-api-access-ljljb" (OuterVolumeSpecName: "kube-api-access-ljljb") pod "b5934576-2058-4ea7-bf91-c8e62d7fc379" (UID: "b5934576-2058-4ea7-bf91-c8e62d7fc379"). InnerVolumeSpecName "kube-api-access-ljljb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:13.907390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.907361    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5934576-2058-4ea7-bf91-c8e62d7fc379-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5934576-2058-4ea7-bf91-c8e62d7fc379" (UID: "b5934576-2058-4ea7-bf91-c8e62d7fc379"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:13.907390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.907378    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-dshm" (OuterVolumeSpecName: "dshm") pod "b5934576-2058-4ea7-bf91-c8e62d7fc379" (UID: "b5934576-2058-4ea7-bf91-c8e62d7fc379"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:13.984454 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:13.984375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:14.006473 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.006444    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.006473 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.006472    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5934576-2058-4ea7-bf91-c8e62d7fc379-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.006632 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.006488    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.006632 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.006497    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5934576-2058-4ea7-bf91-c8e62d7fc379-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.006632 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.006506    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljljb\" (UniqueName: \"kubernetes.io/projected/b5934576-2058-4ea7-bf91-c8e62d7fc379-kube-api-access-ljljb\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.116276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.116252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:26:14.207973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.207936    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-model-cache\") pod \"5cd77d16-0421-47dd-92d0-cace3f795a45\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") "
Sep 09 20:26:14.207973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.207975    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhg24\" (UniqueName: \"kubernetes.io/projected/5cd77d16-0421-47dd-92d0-cace3f795a45-kube-api-access-bhg24\") pod \"5cd77d16-0421-47dd-92d0-cace3f795a45\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") "
Sep 09 20:26:14.208224 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.208006    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-tmp-dir\") pod \"5cd77d16-0421-47dd-92d0-cace3f795a45\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") "
Sep 09 20:26:14.208224 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.208044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd77d16-0421-47dd-92d0-cace3f795a45-tls-certs\") pod \"5cd77d16-0421-47dd-92d0-cace3f795a45\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") "
Sep 09 20:26:14.208224 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.208106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-home\") pod \"5cd77d16-0421-47dd-92d0-cace3f795a45\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") "
Sep 09 20:26:14.208224 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.208134    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-dshm\") pod \"5cd77d16-0421-47dd-92d0-cace3f795a45\" (UID: \"5cd77d16-0421-47dd-92d0-cace3f795a45\") "
Sep 09 20:26:14.208455 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.208227    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cd77d16-0421-47dd-92d0-cace3f795a45" (UID: "5cd77d16-0421-47dd-92d0-cace3f795a45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:14.208455 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.208364    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.208455 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.208370    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cd77d16-0421-47dd-92d0-cace3f795a45" (UID: "5cd77d16-0421-47dd-92d0-cace3f795a45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:14.208455 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.208409    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-home" (OuterVolumeSpecName: "home") pod "5cd77d16-0421-47dd-92d0-cace3f795a45" (UID: "5cd77d16-0421-47dd-92d0-cace3f795a45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:14.210139 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.210118    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd77d16-0421-47dd-92d0-cace3f795a45-kube-api-access-bhg24" (OuterVolumeSpecName: "kube-api-access-bhg24") pod "5cd77d16-0421-47dd-92d0-cace3f795a45" (UID: "5cd77d16-0421-47dd-92d0-cace3f795a45"). InnerVolumeSpecName "kube-api-access-bhg24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:14.210229 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.210209    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd77d16-0421-47dd-92d0-cace3f795a45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cd77d16-0421-47dd-92d0-cace3f795a45" (UID: "5cd77d16-0421-47dd-92d0-cace3f795a45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:14.210270 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.210212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-dshm" (OuterVolumeSpecName: "dshm") pod "5cd77d16-0421-47dd-92d0-cace3f795a45" (UID: "5cd77d16-0421-47dd-92d0-cace3f795a45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:14.222445 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.222419    2578 generic.go:356] "Generic (PLEG): container finished" podID="5cd77d16-0421-47dd-92d0-cace3f795a45" containerID="356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace" exitCode=0
Sep 09 20:26:14.222560 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.222462    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"
Sep 09 20:26:14.222560 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.222494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw" event={"ID":"5cd77d16-0421-47dd-92d0-cace3f795a45","Type":"ContainerDied","Data":"356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace"}
Sep 09 20:26:14.222560 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.222542    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw" event={"ID":"5cd77d16-0421-47dd-92d0-cace3f795a45","Type":"ContainerDied","Data":"01bec6c1e2b881a769d56ac2dc3fa063d05d7da17042d1808d1a6a3977c9e472"}
Sep 09 20:26:14.222560 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.222557    2578 scope.go:117] "RemoveContainer" containerID="356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace"
Sep 09 20:26:14.223865 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.223846    2578 generic.go:356] "Generic (PLEG): container finished" podID="b5934576-2058-4ea7-bf91-c8e62d7fc379" containerID="07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89" exitCode=0
Sep 09 20:26:14.223969 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.223896    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"
Sep 09 20:26:14.224019 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.223894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77" event={"ID":"b5934576-2058-4ea7-bf91-c8e62d7fc379","Type":"ContainerDied","Data":"07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89"}
Sep 09 20:26:14.224072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.224039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77" event={"ID":"b5934576-2058-4ea7-bf91-c8e62d7fc379","Type":"ContainerDied","Data":"6a8f32f9462d64247156387e7090636012bbf28a507ca6b518a94c786ff44fc0"}
Sep 09 20:26:14.230698 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.230685    2578 scope.go:117] "RemoveContainer" containerID="356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace"
Sep 09 20:26:14.231004 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:26:14.230976    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace\": container with ID starting with 356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace not found: ID does not exist" containerID="356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace"
Sep 09 20:26:14.231128 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.231014    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace"} err="failed to get container status \"356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace\": rpc error: code = NotFound desc = could not find container \"356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace\": container with ID starting with 356437258ce4793f81533480f30bf65be6efc0be2236518c17a9e22e92f1bace not found: ID does not exist"
Sep 09 20:26:14.231128 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.231036    2578 scope.go:117] "RemoveContainer" containerID="07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89"
Sep 09 20:26:14.238630 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.238615    2578 scope.go:117] "RemoveContainer" containerID="07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89"
Sep 09 20:26:14.238939 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:26:14.238919    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89\": container with ID starting with 07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89 not found: ID does not exist" containerID="07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89"
Sep 09 20:26:14.239002 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.238944    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89"} err="failed to get container status \"07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89\": rpc error: code = NotFound desc = could not find container \"07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89\": container with ID starting with 07e1b7f5f724d845ac27ac0aed5013baaafab5eef219a46df8dbc7a5c1f50a89 not found: ID does not exist"
Sep 09 20:26:14.249584 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.249564    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"]
Sep 09 20:26:14.255384 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.255365    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw"]
Sep 09 20:26:14.264974 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.264955    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"]
Sep 09 20:26:14.270893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.270875    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-pqc77"]
Sep 09 20:26:14.309637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.309613    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.309637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.309636    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd77d16-0421-47dd-92d0-cace3f795a45-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.309793 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.309644    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.309793 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.309653    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd77d16-0421-47dd-92d0-cace3f795a45-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:14.309793 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:14.309661    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhg24\" (UniqueName: \"kubernetes.io/projected/5cd77d16-0421-47dd-92d0-cace3f795a45-kube-api-access-bhg24\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:26:15.081079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:15.080984    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-zkfrw" podUID="5cd77d16-0421-47dd-92d0-cace3f795a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:26:15.896057 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:15.896019    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd77d16-0421-47dd-92d0-cace3f795a45" path="/var/lib/kubelet/pods/5cd77d16-0421-47dd-92d0-cace3f795a45/volumes"
Sep 09 20:26:15.896460 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:15.896446    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5934576-2058-4ea7-bf91-c8e62d7fc379" path="/var/lib/kubelet/pods/b5934576-2058-4ea7-bf91-c8e62d7fc379/volumes"
Sep 09 20:26:16.476991 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:16.476959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:22.359958 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:22.359923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:22.423168 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:22.423124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:31.660761 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:31.660724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:39.765618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:39.765585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:43.560079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.560044    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"]
Sep 09 20:26:43.560551 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.560376    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5934576-2058-4ea7-bf91-c8e62d7fc379" containerName="main"
Sep 09 20:26:43.560551 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.560389    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5934576-2058-4ea7-bf91-c8e62d7fc379" containerName="main"
Sep 09 20:26:43.560551 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.560410    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd77d16-0421-47dd-92d0-cace3f795a45" containerName="main"
Sep 09 20:26:43.560551 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.560415    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd77d16-0421-47dd-92d0-cace3f795a45" containerName="main"
Sep 09 20:26:43.560551 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.560471    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5934576-2058-4ea7-bf91-c8e62d7fc379" containerName="main"
Sep 09 20:26:43.560551 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.560480    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd77d16-0421-47dd-92d0-cace3f795a45" containerName="main"
Sep 09 20:26:43.563399 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.563381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.566637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.566608    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 09 20:26:43.566808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.566702    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-22988\""
Sep 09 20:26:43.566808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.566735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-b2mps\""
Sep 09 20:26:43.566808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.566751    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 09 20:26:43.566808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.566772    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 09 20:26:43.575614 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.575586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"]
Sep 09 20:26:43.624222 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.624157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:43.657136 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.657101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.657291 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.657147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.657291 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.657167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.657291 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.657216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b5bc6f-513d-46fc-90b7-f46c8662602d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.657291 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.657247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.657461 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.657344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkrzz\" (UniqueName: \"kubernetes.io/projected/53b5bc6f-513d-46fc-90b7-f46c8662602d-kube-api-access-qkrzz\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.758430 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.758395    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.758430 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.758429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.758657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.758456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b5bc6f-513d-46fc-90b7-f46c8662602d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.758657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.758485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.758657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.758545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkrzz\" (UniqueName: \"kubernetes.io/projected/53b5bc6f-513d-46fc-90b7-f46c8662602d-kube-api-access-qkrzz\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.758657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.758594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.758874 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.758801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.758874 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.758850    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.758976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.758906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.760890 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.760864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.761181 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.761160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b5bc6f-513d-46fc-90b7-f46c8662602d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.769691 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.769667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkrzz\" (UniqueName: \"kubernetes.io/projected/53b5bc6f-513d-46fc-90b7-f46c8662602d-kube-api-access-qkrzz\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:43.872706 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:43.872670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:44.146826 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:44.146797    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"]
Sep 09 20:26:44.149470 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:26:44.149446    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b5bc6f_513d_46fc_90b7_f46c8662602d.slice/crio-c412d12df4312115f1cb3ee77a950b464758c7a30822a19cac468a3acd46a96d WatchSource:0}: Error finding container c412d12df4312115f1cb3ee77a950b464758c7a30822a19cac468a3acd46a96d: Status 404 returned error can't find the container with id c412d12df4312115f1cb3ee77a950b464758c7a30822a19cac468a3acd46a96d
Sep 09 20:26:44.354226 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:44.354184    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" event={"ID":"53b5bc6f-513d-46fc-90b7-f46c8662602d","Type":"ContainerStarted","Data":"323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec"}
Sep 09 20:26:44.354440 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:44.354242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" event={"ID":"53b5bc6f-513d-46fc-90b7-f46c8662602d","Type":"ContainerStarted","Data":"c412d12df4312115f1cb3ee77a950b464758c7a30822a19cac468a3acd46a96d"}
Sep 09 20:26:44.354440 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:44.354327    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:44.835726 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:44.835637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:45.360130 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:45.360092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" event={"ID":"53b5bc6f-513d-46fc-90b7-f46c8662602d","Type":"ContainerStarted","Data":"a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713"}
Sep 09 20:26:45.380976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:45.380918    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" podStartSLOduration=2.380901576 podStartE2EDuration="2.380901576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:45.379736205 +0000 UTC m=+1913.968389863" watchObservedRunningTime="2026-09-09 20:26:45.380901576 +0000 UTC m=+1913.969555233"
Sep 09 20:26:45.781719 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:45.781628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:45.873702 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:45.873665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:45.878496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:45.878471    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:53.872802 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:53.872763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:53.885162 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:53.885132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:54.411082 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:54.411047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:26:54.413144 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:26:54.413118    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:27:12.068223 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:12.068183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:20.320790 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.320758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:20.352692 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.352656    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"]
Sep 09 20:27:20.361277 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:27:20.361238    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:27:20.361277 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:27:20.361277 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:27:20.361277 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713" cmd=["/bin/sleep","15"]
Sep 09 20:27:20.361520 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.361291    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" message="" err=<
Sep 09 20:27:20.361520 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:27:20.361520 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:27:20.361520 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:27:20.361520 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:27:20.361303    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:27:20.361520 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:27:20.361520 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:27:20.361520 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="main" containerID="cri-o://a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713"
Sep 09 20:27:20.361520 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.361354    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="main" containerID="cri-o://a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713" gracePeriod=60
Sep 09 20:27:20.399915 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.399871    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8001/health\": dial tcp 10.132.0.46:8001: connect: connection refused"
Sep 09 20:27:20.450454 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.450402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:20.579608 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.579574    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="llm-d-routing-sidecar" containerID="cri-o://323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec" gracePeriod=60
Sep 09 20:27:20.822130 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.822101    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:27:20.877915 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.877883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-dshm\") pod \"53b5bc6f-513d-46fc-90b7-f46c8662602d\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") "
Sep 09 20:27:20.877915 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.877918    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkrzz\" (UniqueName: \"kubernetes.io/projected/53b5bc6f-513d-46fc-90b7-f46c8662602d-kube-api-access-qkrzz\") pod \"53b5bc6f-513d-46fc-90b7-f46c8662602d\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") "
Sep 09 20:27:20.878144 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.877965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-tmp-dir\") pod \"53b5bc6f-513d-46fc-90b7-f46c8662602d\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") "
Sep 09 20:27:20.878144 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.877988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-model-cache\") pod \"53b5bc6f-513d-46fc-90b7-f46c8662602d\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") "
Sep 09 20:27:20.878144 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.878054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-home\") pod \"53b5bc6f-513d-46fc-90b7-f46c8662602d\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") "
Sep 09 20:27:20.878144 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.878076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b5bc6f-513d-46fc-90b7-f46c8662602d-tls-certs\") pod \"53b5bc6f-513d-46fc-90b7-f46c8662602d\" (UID: \"53b5bc6f-513d-46fc-90b7-f46c8662602d\") "
Sep 09 20:27:20.878427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.878299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-model-cache" (OuterVolumeSpecName: "model-cache") pod "53b5bc6f-513d-46fc-90b7-f46c8662602d" (UID: "53b5bc6f-513d-46fc-90b7-f46c8662602d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:20.878427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.878339    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53b5bc6f-513d-46fc-90b7-f46c8662602d" (UID: "53b5bc6f-513d-46fc-90b7-f46c8662602d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:20.878427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.878359    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-home" (OuterVolumeSpecName: "home") pod "53b5bc6f-513d-46fc-90b7-f46c8662602d" (UID: "53b5bc6f-513d-46fc-90b7-f46c8662602d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:20.878580 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.878485    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:27:20.878580 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.878507    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:27:20.878580 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.878520    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:27:20.880394 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.880368    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-dshm" (OuterVolumeSpecName: "dshm") pod "53b5bc6f-513d-46fc-90b7-f46c8662602d" (UID: "53b5bc6f-513d-46fc-90b7-f46c8662602d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:20.880573 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.880550    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b5bc6f-513d-46fc-90b7-f46c8662602d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53b5bc6f-513d-46fc-90b7-f46c8662602d" (UID: "53b5bc6f-513d-46fc-90b7-f46c8662602d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:20.880708 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.880575    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b5bc6f-513d-46fc-90b7-f46c8662602d-kube-api-access-qkrzz" (OuterVolumeSpecName: "kube-api-access-qkrzz") pod "53b5bc6f-513d-46fc-90b7-f46c8662602d" (UID: "53b5bc6f-513d-46fc-90b7-f46c8662602d"). InnerVolumeSpecName "kube-api-access-qkrzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:27:20.979441 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.979405    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b5bc6f-513d-46fc-90b7-f46c8662602d-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:27:20.979441 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.979437    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b5bc6f-513d-46fc-90b7-f46c8662602d-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:27:20.979441 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:20.979447    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkrzz\" (UniqueName: \"kubernetes.io/projected/53b5bc6f-513d-46fc-90b7-f46c8662602d-kube-api-access-qkrzz\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:27:21.518213 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.518173    2578 generic.go:356] "Generic (PLEG): container finished" podID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerID="a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713" exitCode=0
Sep 09 20:27:21.518213 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.518199    2578 generic.go:356] "Generic (PLEG): container finished" podID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerID="323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec" exitCode=0
Sep 09 20:27:21.518749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.518222    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"
Sep 09 20:27:21.518749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.518258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" event={"ID":"53b5bc6f-513d-46fc-90b7-f46c8662602d","Type":"ContainerDied","Data":"a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713"}
Sep 09 20:27:21.518749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.518324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" event={"ID":"53b5bc6f-513d-46fc-90b7-f46c8662602d","Type":"ContainerDied","Data":"323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec"}
Sep 09 20:27:21.518749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.518347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf" event={"ID":"53b5bc6f-513d-46fc-90b7-f46c8662602d","Type":"ContainerDied","Data":"c412d12df4312115f1cb3ee77a950b464758c7a30822a19cac468a3acd46a96d"}
Sep 09 20:27:21.518749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.518346    2578 scope.go:117] "RemoveContainer" containerID="a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713"
Sep 09 20:27:21.527270 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.527250    2578 scope.go:117] "RemoveContainer" containerID="323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec"
Sep 09 20:27:21.537344 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.537326    2578 scope.go:117] "RemoveContainer" containerID="a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713"
Sep 09 20:27:21.537667 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:27:21.537647    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713\": container with ID starting with a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713 not found: ID does not exist" containerID="a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713"
Sep 09 20:27:21.537738 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.537677    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713"} err="failed to get container status \"a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713\": rpc error: code = NotFound desc = could not find container \"a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713\": container with ID starting with a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713 not found: ID does not exist"
Sep 09 20:27:21.537738 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.537704    2578 scope.go:117] "RemoveContainer" containerID="323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec"
Sep 09 20:27:21.538009 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:27:21.537987    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec\": container with ID starting with 323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec not found: ID does not exist" containerID="323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec"
Sep 09 20:27:21.538091 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.538014    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec"} err="failed to get container status \"323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec\": rpc error: code = NotFound desc = could not find container \"323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec\": container with ID starting with 323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec not found: ID does not exist"
Sep 09 20:27:21.538091 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.538029    2578 scope.go:117] "RemoveContainer" containerID="a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713"
Sep 09 20:27:21.538367 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.538331    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713"} err="failed to get container status \"a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713\": rpc error: code = NotFound desc = could not find container \"a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713\": container with ID starting with a03fe21268273d1d4cfcc507d5f7ceb6ea2102fd7ecea7853a67622b38f8d713 not found: ID does not exist"
Sep 09 20:27:21.538367 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.538356    2578 scope.go:117] "RemoveContainer" containerID="323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec"
Sep 09 20:27:21.538827 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.538704    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec"} err="failed to get container status \"323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec\": rpc error: code = NotFound desc = could not find container \"323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec\": container with ID starting with 323c36126ed7e1f15597f55b69d1cceece90a943727c3835a19fdf2a90595aec not found: ID does not exist"
Sep 09 20:27:21.540109 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.540090    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"]
Sep 09 20:27:21.544081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.544059    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-2jpbf"]
Sep 09 20:27:21.896524 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:21.896493    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" path="/var/lib/kubelet/pods/53b5bc6f-513d-46fc-90b7-f46c8662602d/volumes"
Sep 09 20:27:25.143893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:25.143861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:46.810432 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:46.810395    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:46.910977 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:46.910944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:47.015710 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:47.015676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:50.687461 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:50.687418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:51.889189 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:51.889153    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:53.408941 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.408905    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"]
Sep 09 20:27:53.409353 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.409296    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="main"
Sep 09 20:27:53.409353 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.409327    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="main"
Sep 09 20:27:53.409353 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.409340    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="llm-d-routing-sidecar"
Sep 09 20:27:53.409353 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.409346    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="llm-d-routing-sidecar"
Sep 09 20:27:53.409572 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.409408    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="llm-d-routing-sidecar"
Sep 09 20:27:53.409572 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.409419    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b5bc6f-513d-46fc-90b7-f46c8662602d" containerName="main"
Sep 09 20:27:53.412466 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.412445    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.414732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.414703    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 09 20:27:53.414732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.414728    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 09 20:27:53.415756 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.415741    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-zmp5t\""
Sep 09 20:27:53.415879 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.415784    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 09 20:27:53.421444 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.421420    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"]
Sep 09 20:27:53.438483 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.438453    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:53.451695 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.451666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-model-cache\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.451822 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.451712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-home\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.451822 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.451737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.451822 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.451766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b375657-b0aa-417e-9ea5-3d770e41775b-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.451822 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.451793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-dshm\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.451822 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.451816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbqv\" (UniqueName: \"kubernetes.io/projected/1b375657-b0aa-417e-9ea5-3d770e41775b-kube-api-access-rpbqv\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.552962 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.552929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.553146 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.552968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b375657-b0aa-417e-9ea5-3d770e41775b-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.553146 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.553004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-dshm\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.553146 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.553032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbqv\" (UniqueName: \"kubernetes.io/projected/1b375657-b0aa-417e-9ea5-3d770e41775b-kube-api-access-rpbqv\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.553338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.553180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-model-cache\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.553338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.553264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-home\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.553449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.553328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.553534 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.553514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-model-cache\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.553604 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.553585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-home\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.555519 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.555487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-dshm\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.555934 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.555916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b375657-b0aa-417e-9ea5-3d770e41775b-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.560508 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.560484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbqv\" (UniqueName: \"kubernetes.io/projected/1b375657-b0aa-417e-9ea5-3d770e41775b-kube-api-access-rpbqv\") pod \"decom-v1-kserve-6cb797fb75-nzltm\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:53.724244 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:53.724146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:27:54.005923 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:54.005893    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"]
Sep 09 20:27:54.008326 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:27:54.008280    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b375657_b0aa_417e_9ea5_3d770e41775b.slice/crio-7cc943c94db77ca21cb6020d2df882aff0e138bbb160eeef8a89dd4d704a9b90 WatchSource:0}: Error finding container 7cc943c94db77ca21cb6020d2df882aff0e138bbb160eeef8a89dd4d704a9b90: Status 404 returned error can't find the container with id 7cc943c94db77ca21cb6020d2df882aff0e138bbb160eeef8a89dd4d704a9b90
Sep 09 20:27:54.665811 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:54.665774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm" event={"ID":"1b375657-b0aa-417e-9ea5-3d770e41775b","Type":"ContainerStarted","Data":"62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0"}
Sep 09 20:27:54.666177 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:54.665826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm" event={"ID":"1b375657-b0aa-417e-9ea5-3d770e41775b","Type":"ContainerStarted","Data":"7cc943c94db77ca21cb6020d2df882aff0e138bbb160eeef8a89dd4d704a9b90"}
Sep 09 20:27:54.683394 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:54.683343    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm" podStartSLOduration=1.68332496 podStartE2EDuration="1.68332496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:54.681768431 +0000 UTC m=+1983.270422089" watchObservedRunningTime="2026-09-09 20:27:54.68332496 +0000 UTC m=+1983.271978614"
Sep 09 20:27:54.724620 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:27:54.724589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:28:00.746582 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:00.746549    2578 scope.go:117] "RemoveContainer" containerID="0b9431ebdda986a7683759c797bd31f735a08f598b09344404cf4ed29bfcb484"
Sep 09 20:28:01.505025 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.504991    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"]
Sep 09 20:28:01.509688 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.509662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.512052 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.512021    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 09 20:28:01.518581 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.518556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"]
Sep 09 20:28:01.544663 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.544624    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:01.624165 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.624128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqzlk\" (UniqueName: \"kubernetes.io/projected/dce2b130-3505-4ca9-b3c4-534a1120ff4f-kube-api-access-vqzlk\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.624369 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.624176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-model-cache\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.624369 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.624259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tls-certs\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.624369 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.624324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-dshm\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.624369 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.624367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-home\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.624558 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.624399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tmp-dir\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.725419 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.725380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tmp-dir\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.725616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.725434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqzlk\" (UniqueName: \"kubernetes.io/projected/dce2b130-3505-4ca9-b3c4-534a1120ff4f-kube-api-access-vqzlk\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.725616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.725465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-model-cache\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.725616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.725506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tls-certs\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.725616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.725543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-dshm\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.725842 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.725675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-home\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.725908 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.725842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tmp-dir\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.725908 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.725854    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-model-cache\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.726020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.726000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-home\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.727905 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.727879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-dshm\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.728038 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.728021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tls-certs\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.733274 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.733250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqzlk\" (UniqueName: \"kubernetes.io/projected/dce2b130-3505-4ca9-b3c4-534a1120ff4f-kube-api-access-vqzlk\") pod \"decom-v2-kserve-77f867d787-ft5kb\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:01.823072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:01.822973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:02.097522 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:02.097489    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"]
Sep 09 20:28:02.708117 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:02.708080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb" event={"ID":"dce2b130-3505-4ca9-b3c4-534a1120ff4f","Type":"ContainerStarted","Data":"3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce"}
Sep 09 20:28:02.708390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:02.708135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb" event={"ID":"dce2b130-3505-4ca9-b3c4-534a1120ff4f","Type":"ContainerStarted","Data":"629b9ae806e3c2e336ea7cd152d2349c65dfdae79913100ca4fffc0542a6e03e"}
Sep 09 20:28:02.730496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:02.730445    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb" podStartSLOduration=1.730428153 podStartE2EDuration="1.730428153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:02.729690407 +0000 UTC m=+1991.318344129" watchObservedRunningTime="2026-09-09 20:28:02.730428153 +0000 UTC m=+1991.319081810"
Sep 09 20:28:02.823126 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:02.823093    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:03.725081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:03.725051    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:28:03.737547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:03.737516    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:28:04.728665 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:04.728621    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:28:10.435152 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:10.435110    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:10.531187 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:10.531150    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:11.823389 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:11.823355    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:11.835716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:11.835690    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:12.765454 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:12.765415    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:28:16.429505 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:16.429467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:22.131850 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:22.131808    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:22.232140 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:22.232106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:28.988209 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:28.988175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:34.457148 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.457100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:34.516096 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.516020    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:34.826017 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.825932    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"]
Sep 09 20:28:34.829292 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.829271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:34.831756 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.831736    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 09 20:28:34.831891 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.831752    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:28:34.831891 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.831740    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 09 20:28:34.831891 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.831740    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 09 20:28:34.832663 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.832637    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 09 20:28:34.832779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.832647    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-fcrwm\""
Sep 09 20:28:34.841924 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.841900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"]
Sep 09 20:28:34.854546 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:34.854517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:35.014299 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.014261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.014518 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.014359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-model-cache\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.014518 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.014414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-dshm\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.014518 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.014440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6f311b6-2ed5-455e-90ff-172afc6a22d5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.014518 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.014470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-home\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.014668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.014546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.014668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.014647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.014749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.014696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp8hq\" (UniqueName: \"kubernetes.io/projected/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kube-api-access-kp8hq\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115382 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6f311b6-2ed5-455e-90ff-172afc6a22d5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115535 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-home\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115535 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp8hq\" (UniqueName: \"kubernetes.io/projected/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kube-api-access-kp8hq\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115802 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115680    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-model-cache\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115802 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-dshm\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115900 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115803    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-home\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.115958 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.115937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.116253 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.116225    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.116440 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.116265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6f311b6-2ed5-455e-90ff-172afc6a22d5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.116440 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.116375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-model-cache\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.118106 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.118079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-dshm\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.118461 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.118440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.135913 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.135891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp8hq\" (UniqueName: \"kubernetes.io/projected/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kube-api-access-kp8hq\") pod \"tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.140775 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.140753    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:35.201208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.201165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:35.273619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.273577    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:35.415085 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.415059    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"]
Sep 09 20:28:35.417043 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:28:35.417014    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f311b6_2ed5_455e_90ff_172afc6a22d5.slice/crio-c45f70cf89a082188026e7427a482a152e4512172089a5f877ee239ef2bbe9fb WatchSource:0}: Error finding container c45f70cf89a082188026e7427a482a152e4512172089a5f877ee239ef2bbe9fb: Status 404 returned error can't find the container with id c45f70cf89a082188026e7427a482a152e4512172089a5f877ee239ef2bbe9fb
Sep 09 20:28:35.861002 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.860961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" event={"ID":"f6f311b6-2ed5-455e-90ff-172afc6a22d5","Type":"ContainerStarted","Data":"3c89b26fff3bf2d19fc8cdb1b25888a317c04aa3d8afd77d6764a39965592588"}
Sep 09 20:28:35.861508 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:35.861027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" event={"ID":"f6f311b6-2ed5-455e-90ff-172afc6a22d5","Type":"ContainerStarted","Data":"c45f70cf89a082188026e7427a482a152e4512172089a5f877ee239ef2bbe9fb"}
Sep 09 20:28:37.878421 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:37.878366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" event={"ID":"f6f311b6-2ed5-455e-90ff-172afc6a22d5","Type":"ContainerDied","Data":"3c89b26fff3bf2d19fc8cdb1b25888a317c04aa3d8afd77d6764a39965592588"}
Sep 09 20:28:37.879247 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:37.878579    2578 generic.go:356] "Generic (PLEG): container finished" podID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" containerID="3c89b26fff3bf2d19fc8cdb1b25888a317c04aa3d8afd77d6764a39965592588" exitCode=0
Sep 09 20:28:38.886573 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:38.886528    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" event={"ID":"f6f311b6-2ed5-455e-90ff-172afc6a22d5","Type":"ContainerStarted","Data":"38ac8e92cd06ec8aac3f00a799e4a9298df434f9e115499d18401ff549242788"}
Sep 09 20:28:38.911028 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:38.910971    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" podStartSLOduration=2.6203533610000003 podStartE2EDuration="4.910955968s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.290602607s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:38.910328454 +0000 UTC m=+2027.498982109" watchObservedRunningTime="2026-09-09 20:28:38.910955968 +0000 UTC m=+2027.499609622"
Sep 09 20:28:39.142058 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:39.141968    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:45.141293 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:45.141255    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:45.153530 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:45.153504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:45.927600 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:45.927571    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:28:50.136867 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:50.136831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:52.412347 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:52.412285    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"]
Sep 09 20:28:52.433922 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:28:52.433895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:06.723116 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:06.723081    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:07.432903 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.432865    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb" podUID="dce2b130-3505-4ca9-b3c4-534a1120ff4f" containerName="main" containerID="cri-o://3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce" gracePeriod=45
Sep 09 20:29:07.672090 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.672065    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:29:07.785493 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785405    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-home\") pod \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") "
Sep 09 20:29:07.785493 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tmp-dir\") pod \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") "
Sep 09 20:29:07.785493 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tls-certs\") pod \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") "
Sep 09 20:29:07.786035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-model-cache\") pod \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") "
Sep 09 20:29:07.786035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-dshm\") pod \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") "
Sep 09 20:29:07.786035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785625    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqzlk\" (UniqueName: \"kubernetes.io/projected/dce2b130-3505-4ca9-b3c4-534a1120ff4f-kube-api-access-vqzlk\") pod \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\" (UID: \"dce2b130-3505-4ca9-b3c4-534a1120ff4f\") "
Sep 09 20:29:07.786035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785706    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-home" (OuterVolumeSpecName: "home") pod "dce2b130-3505-4ca9-b3c4-534a1120ff4f" (UID: "dce2b130-3505-4ca9-b3c4-534a1120ff4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:07.786035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785799    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dce2b130-3505-4ca9-b3c4-534a1120ff4f" (UID: "dce2b130-3505-4ca9-b3c4-534a1120ff4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:07.786035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785868    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:07.786035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785886    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:07.786035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.785897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "dce2b130-3505-4ca9-b3c4-534a1120ff4f" (UID: "dce2b130-3505-4ca9-b3c4-534a1120ff4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:07.787956 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.787923    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-dshm" (OuterVolumeSpecName: "dshm") pod "dce2b130-3505-4ca9-b3c4-534a1120ff4f" (UID: "dce2b130-3505-4ca9-b3c4-534a1120ff4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:07.788068 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.788008    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dce2b130-3505-4ca9-b3c4-534a1120ff4f" (UID: "dce2b130-3505-4ca9-b3c4-534a1120ff4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:29:07.788068 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.788008    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce2b130-3505-4ca9-b3c4-534a1120ff4f-kube-api-access-vqzlk" (OuterVolumeSpecName: "kube-api-access-vqzlk") pod "dce2b130-3505-4ca9-b3c4-534a1120ff4f" (UID: "dce2b130-3505-4ca9-b3c4-534a1120ff4f"). InnerVolumeSpecName "kube-api-access-vqzlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:29:07.887334 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.887282    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:07.887334 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.887328    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqzlk\" (UniqueName: \"kubernetes.io/projected/dce2b130-3505-4ca9-b3c4-534a1120ff4f-kube-api-access-vqzlk\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:07.887334 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.887338    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce2b130-3505-4ca9-b3c4-534a1120ff4f-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:07.887570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:07.887347    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce2b130-3505-4ca9-b3c4-534a1120ff4f-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:08.015166 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:08.015131    2578 generic.go:356] "Generic (PLEG): container finished" podID="dce2b130-3505-4ca9-b3c4-534a1120ff4f" containerID="3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce" exitCode=0
Sep 09 20:29:08.015350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:08.015171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb" event={"ID":"dce2b130-3505-4ca9-b3c4-534a1120ff4f","Type":"ContainerDied","Data":"3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce"}
Sep 09 20:29:08.015350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:08.015229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb" event={"ID":"dce2b130-3505-4ca9-b3c4-534a1120ff4f","Type":"ContainerDied","Data":"629b9ae806e3c2e336ea7cd152d2349c65dfdae79913100ca4fffc0542a6e03e"}
Sep 09 20:29:08.015350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:08.015184    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"
Sep 09 20:29:08.015350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:08.015250    2578 scope.go:117] "RemoveContainer" containerID="3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce"
Sep 09 20:29:08.026326 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:08.026251    2578 scope.go:117] "RemoveContainer" containerID="3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce"
Sep 09 20:29:08.026735 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:29:08.026710    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce\": container with ID starting with 3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce not found: ID does not exist" containerID="3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce"
Sep 09 20:29:08.026852 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:08.026740    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce"} err="failed to get container status \"3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce\": rpc error: code = NotFound desc = could not find container \"3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce\": container with ID starting with 3707ad29ccfb031af15a4c88aec203d6354c062b314b3b67e14fdd05f80322ce not found: ID does not exist"
Sep 09 20:29:08.034087 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:08.034056    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"]
Sep 09 20:29:08.038437 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:08.038360    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-ft5kb"]
Sep 09 20:29:09.265792 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:09.265760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:09.269743 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:09.269716    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"]
Sep 09 20:29:09.365967 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:09.365935    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:09.465995 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:09.465960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:09.896110 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:09.896070    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce2b130-3505-4ca9-b3c4-534a1120ff4f" path="/var/lib/kubelet/pods/dce2b130-3505-4ca9-b3c4-534a1120ff4f/volumes"
Sep 09 20:29:14.337530 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:14.337437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:17.527389 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.527348    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"]
Sep 09 20:29:17.527920 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.527899    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce2b130-3505-4ca9-b3c4-534a1120ff4f" containerName="main"
Sep 09 20:29:17.527969 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.527926    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce2b130-3505-4ca9-b3c4-534a1120ff4f" containerName="main"
Sep 09 20:29:17.528046 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.528035    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce2b130-3505-4ca9-b3c4-534a1120ff4f" containerName="main"
Sep 09 20:29:17.530996 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.530977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.533639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.533617    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 09 20:29:17.534509 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.534491    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 09 20:29:17.534634 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.534619    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 09 20:29:17.534726 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.534708    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-8r52d\""
Sep 09 20:29:17.540060 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.540039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"]
Sep 09 20:29:17.547899 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.547872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:17.670208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.670174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.670418 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.670217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-model-cache\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.670418 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.670283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-home\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.670418 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.670359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z6ct\" (UniqueName: \"kubernetes.io/projected/4d9836e7-2c85-4721-9db1-bf53b2ef3031-kube-api-access-6z6ct\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.670418 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.670416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-dshm\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.670575 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.670437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tls-certs\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.771787 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.771753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-home\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.771975 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.771798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z6ct\" (UniqueName: \"kubernetes.io/projected/4d9836e7-2c85-4721-9db1-bf53b2ef3031-kube-api-access-6z6ct\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.771975 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.771841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-dshm\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.771975 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.771858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tls-certs\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.771975 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.771891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.771975 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.771914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-model-cache\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.772280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.772254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-home\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.772363 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.772261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-model-cache\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.772363 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.772345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.774340 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.774300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-dshm\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.774652 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.774630    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tls-certs\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.784140 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.784076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z6ct\" (UniqueName: \"kubernetes.io/projected/4d9836e7-2c85-4721-9db1-bf53b2ef3031-kube-api-access-6z6ct\") pod \"leave-v1-kserve-65684d4f5-l88bz\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:17.841731 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:17.841689    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:18.120624 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:18.120597    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"]
Sep 09 20:29:18.122773 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:29:18.122731    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9836e7_2c85_4721_9db1_bf53b2ef3031.slice/crio-18a9864c98cfd03d839f70cb26748e0b8da1897e8a7763e897d50634001560e9 WatchSource:0}: Error finding container 18a9864c98cfd03d839f70cb26748e0b8da1897e8a7763e897d50634001560e9: Status 404 returned error can't find the container with id 18a9864c98cfd03d839f70cb26748e0b8da1897e8a7763e897d50634001560e9
Sep 09 20:29:19.069448 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:19.069414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz" event={"ID":"4d9836e7-2c85-4721-9db1-bf53b2ef3031","Type":"ContainerStarted","Data":"057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49"}
Sep 09 20:29:19.069812 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:19.069470    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz" event={"ID":"4d9836e7-2c85-4721-9db1-bf53b2ef3031","Type":"ContainerStarted","Data":"18a9864c98cfd03d839f70cb26748e0b8da1897e8a7763e897d50634001560e9"}
Sep 09 20:29:19.089300 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:19.089250    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz" podStartSLOduration=2.089234384 podStartE2EDuration="2.089234384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:29:19.08705147 +0000 UTC m=+2067.675705128" watchObservedRunningTime="2026-09-09 20:29:19.089234384 +0000 UTC m=+2067.677888041"
Sep 09 20:29:19.842006 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:19.841967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:24.294416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.294345    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm" podUID="1b375657-b0aa-417e-9ea5-3d770e41775b" containerName="main" containerID="cri-o://62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0" gracePeriod=45
Sep 09 20:29:24.552181 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.552119    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:29:24.628980 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.628943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b375657-b0aa-417e-9ea5-3d770e41775b-tls-certs\") pod \"1b375657-b0aa-417e-9ea5-3d770e41775b\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") "
Sep 09 20:29:24.629172 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.628995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-tmp-dir\") pod \"1b375657-b0aa-417e-9ea5-3d770e41775b\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") "
Sep 09 20:29:24.629172 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.629014    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-model-cache\") pod \"1b375657-b0aa-417e-9ea5-3d770e41775b\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") "
Sep 09 20:29:24.629172 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.629056    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpbqv\" (UniqueName: \"kubernetes.io/projected/1b375657-b0aa-417e-9ea5-3d770e41775b-kube-api-access-rpbqv\") pod \"1b375657-b0aa-417e-9ea5-3d770e41775b\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") "
Sep 09 20:29:24.629172 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.629078    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-home\") pod \"1b375657-b0aa-417e-9ea5-3d770e41775b\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") "
Sep 09 20:29:24.629172 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.629136    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-dshm\") pod \"1b375657-b0aa-417e-9ea5-3d770e41775b\" (UID: \"1b375657-b0aa-417e-9ea5-3d770e41775b\") "
Sep 09 20:29:24.629483 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.629356    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b375657-b0aa-417e-9ea5-3d770e41775b" (UID: "1b375657-b0aa-417e-9ea5-3d770e41775b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:24.629483 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.629379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b375657-b0aa-417e-9ea5-3d770e41775b" (UID: "1b375657-b0aa-417e-9ea5-3d770e41775b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:24.629483 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.629448    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-home" (OuterVolumeSpecName: "home") pod "1b375657-b0aa-417e-9ea5-3d770e41775b" (UID: "1b375657-b0aa-417e-9ea5-3d770e41775b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:24.629483 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.629465    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:24.631292 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.631267    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-dshm" (OuterVolumeSpecName: "dshm") pod "1b375657-b0aa-417e-9ea5-3d770e41775b" (UID: "1b375657-b0aa-417e-9ea5-3d770e41775b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:24.631416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.631289    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b375657-b0aa-417e-9ea5-3d770e41775b-kube-api-access-rpbqv" (OuterVolumeSpecName: "kube-api-access-rpbqv") pod "1b375657-b0aa-417e-9ea5-3d770e41775b" (UID: "1b375657-b0aa-417e-9ea5-3d770e41775b"). InnerVolumeSpecName "kube-api-access-rpbqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:29:24.631416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.631324    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b375657-b0aa-417e-9ea5-3d770e41775b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b375657-b0aa-417e-9ea5-3d770e41775b" (UID: "1b375657-b0aa-417e-9ea5-3d770e41775b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:29:24.730063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.730028    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:24.730063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.730061    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpbqv\" (UniqueName: \"kubernetes.io/projected/1b375657-b0aa-417e-9ea5-3d770e41775b-kube-api-access-rpbqv\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:24.730262 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.730073    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:24.730262 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.730082    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b375657-b0aa-417e-9ea5-3d770e41775b-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:24.730262 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:24.730090    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b375657-b0aa-417e-9ea5-3d770e41775b-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:29:25.097891 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.097854    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b375657-b0aa-417e-9ea5-3d770e41775b" containerID="62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0" exitCode=0
Sep 09 20:29:25.098127 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.097923    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"
Sep 09 20:29:25.098127 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.097938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm" event={"ID":"1b375657-b0aa-417e-9ea5-3d770e41775b","Type":"ContainerDied","Data":"62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0"}
Sep 09 20:29:25.098127 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.098006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm" event={"ID":"1b375657-b0aa-417e-9ea5-3d770e41775b","Type":"ContainerDied","Data":"7cc943c94db77ca21cb6020d2df882aff0e138bbb160eeef8a89dd4d704a9b90"}
Sep 09 20:29:25.098127 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.098031    2578 scope.go:117] "RemoveContainer" containerID="62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0"
Sep 09 20:29:25.106988 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.106973    2578 scope.go:117] "RemoveContainer" containerID="62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0"
Sep 09 20:29:25.107242 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:29:25.107223    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0\": container with ID starting with 62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0 not found: ID does not exist" containerID="62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0"
Sep 09 20:29:25.107294 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.107249    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0"} err="failed to get container status \"62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0\": rpc error: code = NotFound desc = could not find container \"62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0\": container with ID starting with 62b9da28f4803015a99a794f7aed660e9c6f3ee1f629fb942624f7d46aeb52e0 not found: ID does not exist"
Sep 09 20:29:25.118755 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.118732    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"]
Sep 09 20:29:25.122414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.122393    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-nzltm"]
Sep 09 20:29:25.896940 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:25.896904    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b375657-b0aa-417e-9ea5-3d770e41775b" path="/var/lib/kubelet/pods/1b375657-b0aa-417e-9ea5-3d770e41775b/volumes"
Sep 09 20:29:26.196676 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:26.196570    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:27.842580 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:27.842547    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:27.855169 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:27.855143    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:28.123533 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:28.123440    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:29:31.523006 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.522974    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"]
Sep 09 20:29:31.523416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.523319    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b375657-b0aa-417e-9ea5-3d770e41775b" containerName="main"
Sep 09 20:29:31.523416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.523335    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b375657-b0aa-417e-9ea5-3d770e41775b" containerName="main"
Sep 09 20:29:31.523416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.523413    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b375657-b0aa-417e-9ea5-3d770e41775b" containerName="main"
Sep 09 20:29:31.525225 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.525209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.527775 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.527755    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 09 20:29:31.536049 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.536029    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"]
Sep 09 20:29:31.576894 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.576864    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:31.689319 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.689279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.689509 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.689337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-model-cache\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.689509 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.689364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.689509 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.689383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmc5\" (UniqueName: \"kubernetes.io/projected/b96e9b41-fbaa-433e-b966-3613fe6a5c64-kube-api-access-dbmc5\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.689509 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.689409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-home\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.689509 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.689436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-dshm\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.790986 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.790902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-model-cache\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.790986 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.790938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.790986 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.790962    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmc5\" (UniqueName: \"kubernetes.io/projected/b96e9b41-fbaa-433e-b966-3613fe6a5c64-kube-api-access-dbmc5\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.791286 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.790999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-home\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.791286 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.791024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-dshm\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.791286 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.791127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.791481 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.791390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.791481 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.791406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-model-cache\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.791481 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.791466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-home\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.793434 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.793407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-dshm\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.793708 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.793686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.798449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.798422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmc5\" (UniqueName: \"kubernetes.io/projected/b96e9b41-fbaa-433e-b966-3613fe6a5c64-kube-api-access-dbmc5\") pod \"leave-v2-kserve-556b4dddf9-m2z4l\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:31.836463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:31.836428    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:32.118489 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:32.118468    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"]
Sep 09 20:29:32.121192 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:29:32.121161    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96e9b41_fbaa_433e_b966_3613fe6a5c64.slice/crio-9a9881b5269294ba959a3d48a8fabaf219d4870eeb8611be1bf72082e341f538 WatchSource:0}: Error finding container 9a9881b5269294ba959a3d48a8fabaf219d4870eeb8611be1bf72082e341f538: Status 404 returned error can't find the container with id 9a9881b5269294ba959a3d48a8fabaf219d4870eeb8611be1bf72082e341f538
Sep 09 20:29:32.131910 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:32.131884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l" event={"ID":"b96e9b41-fbaa-433e-b966-3613fe6a5c64","Type":"ContainerStarted","Data":"9a9881b5269294ba959a3d48a8fabaf219d4870eeb8611be1bf72082e341f538"}
Sep 09 20:29:33.137508 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:33.137474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l" event={"ID":"b96e9b41-fbaa-433e-b966-3613fe6a5c64","Type":"ContainerStarted","Data":"3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2"}
Sep 09 20:29:33.155534 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:33.155484    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l" podStartSLOduration=2.155469656 podStartE2EDuration="2.155469656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:29:33.153901717 +0000 UTC m=+2081.742555373" watchObservedRunningTime="2026-09-09 20:29:33.155469656 +0000 UTC m=+2081.744123313"
Sep 09 20:29:33.837194 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:33.837151    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:41.836840 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:41.836798    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:41.849205 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:41.849171    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:42.190901 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:42.190862    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:29:51.193443 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:51.193401    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:51.997330 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:51.997279    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:29:52.001790 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:52.001765    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:29:56.931793 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:56.931753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:57.040755 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:57.040696    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:57.704323 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:57.704283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:57.804939 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:57.804892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:57.906095 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:29:57.906060    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:02.689271 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:02.689238    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"]
Sep 09 20:30:02.698811 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:02.698779    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"]
Sep 09 20:30:02.738208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:02.738152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:02.793896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:02.793853    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:03.830982 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:03.830948    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:10.522588 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.522554    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"]
Sep 09 20:30:10.527186 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.527161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.529751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.529727    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 09 20:30:10.530642 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.530617    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 09 20:30:10.530770 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.530617    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-5qldm\""
Sep 09 20:30:10.530770 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.530678    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 09 20:30:10.537787 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.537762    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"]
Sep 09 20:30:10.558678 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.558627    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:10.638269 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.638230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-dshm\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.638479 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.638298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxr7\" (UniqueName: \"kubernetes.io/projected/98678eff-e74f-49da-9808-a722fab2847d-kube-api-access-njxr7\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.638479 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.638359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.638479 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.638400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.638610 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.638479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-home\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.638610 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.638532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98678eff-e74f-49da-9808-a722fab2847d-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.739427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.739388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98678eff-e74f-49da-9808-a722fab2847d-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.739427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.739431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-dshm\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.739674 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.739473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxr7\" (UniqueName: \"kubernetes.io/projected/98678eff-e74f-49da-9808-a722fab2847d-kube-api-access-njxr7\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.739674 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.739503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.739674 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.739532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.739674 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.739559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-home\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.740341 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.740273    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.746701 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.740473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-home\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.746701 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.740648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.747169 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.747134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-dshm\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.747696 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.747674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98678eff-e74f-49da-9808-a722fab2847d-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.749441 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.749420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxr7\" (UniqueName: \"kubernetes.io/projected/98678eff-e74f-49da-9808-a722fab2847d-kube-api-access-njxr7\") pod \"tri-v1-kserve-79f8f98bc5-678pr\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:10.839232 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:10.839126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:11.117483 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:11.117449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"]
Sep 09 20:30:11.315454 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:11.315411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr" event={"ID":"98678eff-e74f-49da-9808-a722fab2847d","Type":"ContainerStarted","Data":"279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d"}
Sep 09 20:30:11.315648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:11.315480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr" event={"ID":"98678eff-e74f-49da-9808-a722fab2847d","Type":"ContainerStarted","Data":"75110dee0c32f1b2dc33ea418bd309b0cf96be22f9901fc8badec0e80492d86e"}
Sep 09 20:30:11.333468 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:11.333417    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr" podStartSLOduration=1.333399207 podStartE2EDuration="1.333399207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:11.332393288 +0000 UTC m=+2119.921046956" watchObservedRunningTime="2026-09-09 20:30:11.333399207 +0000 UTC m=+2119.922052864"
Sep 09 20:30:11.839415 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:11.839362    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:15.665248 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:15.665213    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:17.716979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:17.716915    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz" podUID="4d9836e7-2c85-4721-9db1-bf53b2ef3031" containerName="main" containerID="cri-o://057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49" gracePeriod=45
Sep 09 20:30:17.721497 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:17.721456    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l" podUID="b96e9b41-fbaa-433e-b966-3613fe6a5c64" containerName="main" containerID="cri-o://3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2" gracePeriod=45
Sep 09 20:30:17.969778 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:17.969715    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:30:17.999526 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:17.999492    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:30:18.100795 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.100764    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tmp-dir\") pod \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") "
Sep 09 20:30:18.100795 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.100800    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-dshm\") pod \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") "
Sep 09 20:30:18.101079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.100829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tls-certs\") pod \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") "
Sep 09 20:30:18.101079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.100857    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-model-cache\") pod \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") "
Sep 09 20:30:18.101079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.100899    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-dshm\") pod \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") "
Sep 09 20:30:18.101079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.100913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-home\") pod \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") "
Sep 09 20:30:18.101079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.100938    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z6ct\" (UniqueName: \"kubernetes.io/projected/4d9836e7-2c85-4721-9db1-bf53b2ef3031-kube-api-access-6z6ct\") pod \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") "
Sep 09 20:30:18.101079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.100970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tmp-dir\") pod \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\" (UID: \"4d9836e7-2c85-4721-9db1-bf53b2ef3031\") "
Sep 09 20:30:18.101079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101010    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-model-cache\") pod \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") "
Sep 09 20:30:18.101079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tls-certs\") pod \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") "
Sep 09 20:30:18.101079 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101081    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbmc5\" (UniqueName: \"kubernetes.io/projected/b96e9b41-fbaa-433e-b966-3613fe6a5c64-kube-api-access-dbmc5\") pod \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") "
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b96e9b41-fbaa-433e-b966-3613fe6a5c64" (UID: "b96e9b41-fbaa-433e-b966-3613fe6a5c64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-home\") pod \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\" (UID: \"b96e9b41-fbaa-433e-b966-3613fe6a5c64\") "
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-home" (OuterVolumeSpecName: "home") pod "4d9836e7-2c85-4721-9db1-bf53b2ef3031" (UID: "4d9836e7-2c85-4721-9db1-bf53b2ef3031"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d9836e7-2c85-4721-9db1-bf53b2ef3031" (UID: "4d9836e7-2c85-4721-9db1-bf53b2ef3031"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101284    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-model-cache" (OuterVolumeSpecName: "model-cache") pod "b96e9b41-fbaa-433e-b966-3613fe6a5c64" (UID: "b96e9b41-fbaa-433e-b966-3613fe6a5c64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101483    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101491    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d9836e7-2c85-4721-9db1-bf53b2ef3031" (UID: "4d9836e7-2c85-4721-9db1-bf53b2ef3031"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101504    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101518    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.101637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101531    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.102142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.101674    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-home" (OuterVolumeSpecName: "home") pod "b96e9b41-fbaa-433e-b966-3613fe6a5c64" (UID: "b96e9b41-fbaa-433e-b966-3613fe6a5c64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:18.103507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.103477    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96e9b41-fbaa-433e-b966-3613fe6a5c64-kube-api-access-dbmc5" (OuterVolumeSpecName: "kube-api-access-dbmc5") pod "b96e9b41-fbaa-433e-b966-3613fe6a5c64" (UID: "b96e9b41-fbaa-433e-b966-3613fe6a5c64"). InnerVolumeSpecName "kube-api-access-dbmc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:18.103632 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.103553    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d9836e7-2c85-4721-9db1-bf53b2ef3031" (UID: "4d9836e7-2c85-4721-9db1-bf53b2ef3031"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:30:18.103695 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.103641    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-dshm" (OuterVolumeSpecName: "dshm") pod "b96e9b41-fbaa-433e-b966-3613fe6a5c64" (UID: "b96e9b41-fbaa-433e-b966-3613fe6a5c64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:18.103695 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.103653    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-dshm" (OuterVolumeSpecName: "dshm") pod "4d9836e7-2c85-4721-9db1-bf53b2ef3031" (UID: "4d9836e7-2c85-4721-9db1-bf53b2ef3031"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:18.103695 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.103661    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9836e7-2c85-4721-9db1-bf53b2ef3031-kube-api-access-6z6ct" (OuterVolumeSpecName: "kube-api-access-6z6ct") pod "4d9836e7-2c85-4721-9db1-bf53b2ef3031" (UID: "4d9836e7-2c85-4721-9db1-bf53b2ef3031"). InnerVolumeSpecName "kube-api-access-6z6ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:18.103831 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.103747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b96e9b41-fbaa-433e-b966-3613fe6a5c64" (UID: "b96e9b41-fbaa-433e-b966-3613fe6a5c64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:30:18.202153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.202116    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.202153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.202145    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.202375 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.202154    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.202375 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.202178    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b96e9b41-fbaa-433e-b966-3613fe6a5c64-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.202375 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.202188    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z6ct\" (UniqueName: \"kubernetes.io/projected/4d9836e7-2c85-4721-9db1-bf53b2ef3031-kube-api-access-6z6ct\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.202375 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.202196    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d9836e7-2c85-4721-9db1-bf53b2ef3031-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.202375 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.202204    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b96e9b41-fbaa-433e-b966-3613fe6a5c64-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.202375 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.202212    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmc5\" (UniqueName: \"kubernetes.io/projected/b96e9b41-fbaa-433e-b966-3613fe6a5c64-kube-api-access-dbmc5\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:18.348594 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.348554    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d9836e7-2c85-4721-9db1-bf53b2ef3031" containerID="057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49" exitCode=0
Sep 09 20:30:18.348789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.348604    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"
Sep 09 20:30:18.348789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.348648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz" event={"ID":"4d9836e7-2c85-4721-9db1-bf53b2ef3031","Type":"ContainerDied","Data":"057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49"}
Sep 09 20:30:18.348789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.348714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz" event={"ID":"4d9836e7-2c85-4721-9db1-bf53b2ef3031","Type":"ContainerDied","Data":"18a9864c98cfd03d839f70cb26748e0b8da1897e8a7763e897d50634001560e9"}
Sep 09 20:30:18.348789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.348744    2578 scope.go:117] "RemoveContainer" containerID="057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49"
Sep 09 20:30:18.350086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.350064    2578 generic.go:356] "Generic (PLEG): container finished" podID="b96e9b41-fbaa-433e-b966-3613fe6a5c64" containerID="3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2" exitCode=0
Sep 09 20:30:18.350189 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.350115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l" event={"ID":"b96e9b41-fbaa-433e-b966-3613fe6a5c64","Type":"ContainerDied","Data":"3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2"}
Sep 09 20:30:18.350189 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.350122    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"
Sep 09 20:30:18.350189 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.350153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l" event={"ID":"b96e9b41-fbaa-433e-b966-3613fe6a5c64","Type":"ContainerDied","Data":"9a9881b5269294ba959a3d48a8fabaf219d4870eeb8611be1bf72082e341f538"}
Sep 09 20:30:18.357318 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.357286    2578 scope.go:117] "RemoveContainer" containerID="057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49"
Sep 09 20:30:18.357577 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:30:18.357560    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49\": container with ID starting with 057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49 not found: ID does not exist" containerID="057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49"
Sep 09 20:30:18.357644 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.357586    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49"} err="failed to get container status \"057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49\": rpc error: code = NotFound desc = could not find container \"057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49\": container with ID starting with 057fcb144448c13da5abbb68683c45024c7d6b1081af626aaab349195d201f49 not found: ID does not exist"
Sep 09 20:30:18.357644 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.357601    2578 scope.go:117] "RemoveContainer" containerID="3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2"
Sep 09 20:30:18.365734 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.365715    2578 scope.go:117] "RemoveContainer" containerID="3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2"
Sep 09 20:30:18.365983 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:30:18.365966    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2\": container with ID starting with 3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2 not found: ID does not exist" containerID="3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2"
Sep 09 20:30:18.366030 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.365988    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2"} err="failed to get container status \"3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2\": rpc error: code = NotFound desc = could not find container \"3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2\": container with ID starting with 3c79142df387f5422ae3b6779e1712afd219ecdd6d9b6e2ea1d07c5266b305d2 not found: ID does not exist"
Sep 09 20:30:18.376476 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.376455    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"]
Sep 09 20:30:18.381116 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.381097    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-l88bz"]
Sep 09 20:30:18.394898 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.394869    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"]
Sep 09 20:30:18.399034 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:18.399010    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-m2z4l"]
Sep 09 20:30:19.897226 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:19.897188    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9836e7-2c85-4721-9db1-bf53b2ef3031" path="/var/lib/kubelet/pods/4d9836e7-2c85-4721-9db1-bf53b2ef3031/volumes"
Sep 09 20:30:19.897668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:19.897653    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96e9b41-fbaa-433e-b966-3613fe6a5c64" path="/var/lib/kubelet/pods/b96e9b41-fbaa-433e-b966-3613fe6a5c64/volumes"
Sep 09 20:30:20.840021 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:20.839981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:20.852342 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:20.852294    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:21.377891 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:21.377857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:30:27.541502 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.541470    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"]
Sep 09 20:30:27.541888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.541817    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96e9b41-fbaa-433e-b966-3613fe6a5c64" containerName="main"
Sep 09 20:30:27.541888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.541828    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96e9b41-fbaa-433e-b966-3613fe6a5c64" containerName="main"
Sep 09 20:30:27.541888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.541839    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9836e7-2c85-4721-9db1-bf53b2ef3031" containerName="main"
Sep 09 20:30:27.541888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.541844    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9836e7-2c85-4721-9db1-bf53b2ef3031" containerName="main"
Sep 09 20:30:27.542039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.541896    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96e9b41-fbaa-433e-b966-3613fe6a5c64" containerName="main"
Sep 09 20:30:27.542039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.541909    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9836e7-2c85-4721-9db1-bf53b2ef3031" containerName="main"
Sep 09 20:30:27.544637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.544620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.547180 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.547160    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 09 20:30:27.554827 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.554800    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"]
Sep 09 20:30:27.564719 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.564690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:27.685695 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.685661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjz7\" (UniqueName: \"kubernetes.io/projected/837212d5-ea23-4ffa-8429-bd5914ce76f0-kube-api-access-ndjz7\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.685841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.685717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.685841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.685739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-model-cache\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.685841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.685779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-dshm\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.685841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.685794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837212d5-ea23-4ffa-8429-bd5914ce76f0-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.685981 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.685898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-home\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.786548 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.786517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-home\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.786723 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.786554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjz7\" (UniqueName: \"kubernetes.io/projected/837212d5-ea23-4ffa-8429-bd5914ce76f0-kube-api-access-ndjz7\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.786766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.786729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.786766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.786752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-model-cache\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.786859 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.786783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-dshm\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.786859 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.786798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837212d5-ea23-4ffa-8429-bd5914ce76f0-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.786964 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.786922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-home\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.787087 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.787066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-model-cache\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.787192 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.787171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.789084 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.789062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-dshm\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.789337 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.789298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837212d5-ea23-4ffa-8429-bd5914ce76f0-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.794499 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.794449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjz7\" (UniqueName: \"kubernetes.io/projected/837212d5-ea23-4ffa-8429-bd5914ce76f0-kube-api-access-ndjz7\") pod \"tri-v2-kserve-dcb554bf5-9mm6g\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:27.855004 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:27.854981    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:28.146215 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:28.146189    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"]
Sep 09 20:30:28.148045 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:30:28.148009    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837212d5_ea23_4ffa_8429_bd5914ce76f0.slice/crio-d12374c95dbb524a7c258a06518291c3d67d0d4feaab46e7eebb7f4d54181d69 WatchSource:0}: Error finding container d12374c95dbb524a7c258a06518291c3d67d0d4feaab46e7eebb7f4d54181d69: Status 404 returned error can't find the container with id d12374c95dbb524a7c258a06518291c3d67d0d4feaab46e7eebb7f4d54181d69
Sep 09 20:30:28.403682 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:28.403480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g" event={"ID":"837212d5-ea23-4ffa-8429-bd5914ce76f0","Type":"ContainerStarted","Data":"d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b"}
Sep 09 20:30:28.403682 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:28.403579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g" event={"ID":"837212d5-ea23-4ffa-8429-bd5914ce76f0","Type":"ContainerStarted","Data":"d12374c95dbb524a7c258a06518291c3d67d0d4feaab46e7eebb7f4d54181d69"}
Sep 09 20:30:28.425030 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:28.424981    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g" podStartSLOduration=1.424965569 podStartE2EDuration="1.424965569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:28.422533238 +0000 UTC m=+2137.011186895" watchObservedRunningTime="2026-09-09 20:30:28.424965569 +0000 UTC m=+2137.013619225"
Sep 09 20:30:28.855987 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:28.855953    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:36.526014 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.525972    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"]
Sep 09 20:30:36.529973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.529953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.532663 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.532641    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 09 20:30:36.539397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.539370    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"]
Sep 09 20:30:36.555197 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.555173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-home\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.555331 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.555211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzp5\" (UniqueName: \"kubernetes.io/projected/f45c1efb-054c-4607-9c83-6c811cdfa960-kube-api-access-ghzp5\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.555331 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.555277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.555414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.555357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f45c1efb-054c-4607-9c83-6c811cdfa960-tls-certs\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.555414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.555382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-model-cache\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.555414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.555399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-dshm\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.577690 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.577665    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:36.656595 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.656563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f45c1efb-054c-4607-9c83-6c811cdfa960-tls-certs\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.656756 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.656610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-model-cache\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.656756 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.656635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-dshm\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.656756 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.656698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-home\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.656756 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.656735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzp5\" (UniqueName: \"kubernetes.io/projected/f45c1efb-054c-4607-9c83-6c811cdfa960-kube-api-access-ghzp5\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.656975 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.656771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.657102 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.657077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-model-cache\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.657167 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.657142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-home\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.657224 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.657182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.658952 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.658934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-dshm\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.659354 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.659334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f45c1efb-054c-4607-9c83-6c811cdfa960-tls-certs\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.668221 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.668163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzp5\" (UniqueName: \"kubernetes.io/projected/f45c1efb-054c-4607-9c83-6c811cdfa960-kube-api-access-ghzp5\") pod \"tri-v3-kserve-798c88c66c-dgt6n\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:36.843685 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:36.843649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:37.116907 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:37.116872    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"]
Sep 09 20:30:37.120148 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:30:37.120122    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45c1efb_054c_4607_9c83_6c811cdfa960.slice/crio-903f7f85df108733a854eef5940b61259f167d9361bf688c48c60421e67ae8b0 WatchSource:0}: Error finding container 903f7f85df108733a854eef5940b61259f167d9361bf688c48c60421e67ae8b0: Status 404 returned error can't find the container with id 903f7f85df108733a854eef5940b61259f167d9361bf688c48c60421e67ae8b0
Sep 09 20:30:37.446637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:37.446550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n" event={"ID":"f45c1efb-054c-4607-9c83-6c811cdfa960","Type":"ContainerStarted","Data":"783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8"}
Sep 09 20:30:37.446637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:37.446605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n" event={"ID":"f45c1efb-054c-4607-9c83-6c811cdfa960","Type":"ContainerStarted","Data":"903f7f85df108733a854eef5940b61259f167d9361bf688c48c60421e67ae8b0"}
Sep 09 20:30:37.482980 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:37.482926    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n" podStartSLOduration=1.482912136 podStartE2EDuration="1.482912136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:37.482355555 +0000 UTC m=+2146.071009213" watchObservedRunningTime="2026-09-09 20:30:37.482912136 +0000 UTC m=+2146.071565829"
Sep 09 20:30:37.844161 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:37.844125    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:37.856135 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:37.856098    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:37.870007 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:37.869981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:38.150452 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.150356    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"]
Sep 09 20:30:38.152502 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.152457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:38.160243 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:30:38.160210    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:30:38.160243 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:30:38.160243 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:30:38.160243 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="38ac8e92cd06ec8aac3f00a799e4a9298df434f9e115499d18401ff549242788" cmd=["/bin/sleep","15"]
Sep 09 20:30:38.160491 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.160259    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" message="" err=<
Sep 09 20:30:38.160491 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:30:38.160491 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:30:38.160491 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:30:38.160491 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:30:38.160273    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:30:38.160491 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:30:38.160491 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:30:38.160491 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" podUID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" containerName="main" containerID="cri-o://38ac8e92cd06ec8aac3f00a799e4a9298df434f9e115499d18401ff549242788"
Sep 09 20:30:38.160491 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.160393    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" podUID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" containerName="main" containerID="cri-o://38ac8e92cd06ec8aac3f00a799e4a9298df434f9e115499d18401ff549242788" gracePeriod=60
Sep 09 20:30:38.237764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.237725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:38.454292 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.454260    2578 generic.go:356] "Generic (PLEG): container finished" podID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" containerID="38ac8e92cd06ec8aac3f00a799e4a9298df434f9e115499d18401ff549242788" exitCode=0
Sep 09 20:30:38.454429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.454348    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" event={"ID":"f6f311b6-2ed5-455e-90ff-172afc6a22d5","Type":"ContainerDied","Data":"38ac8e92cd06ec8aac3f00a799e4a9298df434f9e115499d18401ff549242788"}
Sep 09 20:30:38.466027 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.465998    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:30:38.514861 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.514830    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:30:38.574577 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.574548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-home\") pod \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") "
Sep 09 20:30:38.574779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.574601    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tls-certs\") pod \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") "
Sep 09 20:30:38.574779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.574625    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-dshm\") pod \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") "
Sep 09 20:30:38.574779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.574652    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp8hq\" (UniqueName: \"kubernetes.io/projected/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kube-api-access-kp8hq\") pod \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") "
Sep 09 20:30:38.574779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.574702    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-model-cache\") pod \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") "
Sep 09 20:30:38.574779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.574763    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6f311b6-2ed5-455e-90ff-172afc6a22d5-cabundle-cert\") pod \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") "
Sep 09 20:30:38.575072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.574803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tmp-dir\") pod \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") "
Sep 09 20:30:38.575072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.574883    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-home" (OuterVolumeSpecName: "home") pod "f6f311b6-2ed5-455e-90ff-172afc6a22d5" (UID: "f6f311b6-2ed5-455e-90ff-172afc6a22d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:38.575072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.574994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6f311b6-2ed5-455e-90ff-172afc6a22d5" (UID: "f6f311b6-2ed5-455e-90ff-172afc6a22d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:38.575072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.575006    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kserve-provision-location\") pod \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\" (UID: \"f6f311b6-2ed5-455e-90ff-172afc6a22d5\") "
Sep 09 20:30:38.575272 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.575069    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6f311b6-2ed5-455e-90ff-172afc6a22d5" (UID: "f6f311b6-2ed5-455e-90ff-172afc6a22d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:38.575272 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.575188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f311b6-2ed5-455e-90ff-172afc6a22d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f6f311b6-2ed5-455e-90ff-172afc6a22d5" (UID: "f6f311b6-2ed5-455e-90ff-172afc6a22d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:38.575416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.575384    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:38.575416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.575404    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6f311b6-2ed5-455e-90ff-172afc6a22d5-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:38.575518 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.575419    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:38.575518 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.575431    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:38.576998 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.576970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6f311b6-2ed5-455e-90ff-172afc6a22d5" (UID: "f6f311b6-2ed5-455e-90ff-172afc6a22d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:30:38.577111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.576999    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-dshm" (OuterVolumeSpecName: "dshm") pod "f6f311b6-2ed5-455e-90ff-172afc6a22d5" (UID: "f6f311b6-2ed5-455e-90ff-172afc6a22d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:38.577111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.577075    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kube-api-access-kp8hq" (OuterVolumeSpecName: "kube-api-access-kp8hq") pod "f6f311b6-2ed5-455e-90ff-172afc6a22d5" (UID: "f6f311b6-2ed5-455e-90ff-172afc6a22d5"). InnerVolumeSpecName "kube-api-access-kp8hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:38.599222 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.599188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6f311b6-2ed5-455e-90ff-172afc6a22d5" (UID: "f6f311b6-2ed5-455e-90ff-172afc6a22d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:38.676157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.676130    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:38.676157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.676153    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6f311b6-2ed5-455e-90ff-172afc6a22d5-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:38.676157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.676163    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6f311b6-2ed5-455e-90ff-172afc6a22d5-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:38.676378 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:38.676171    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp8hq\" (UniqueName: \"kubernetes.io/projected/f6f311b6-2ed5-455e-90ff-172afc6a22d5-kube-api-access-kp8hq\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:30:39.461506 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:39.461475    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"
Sep 09 20:30:39.461892 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:39.461513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8" event={"ID":"f6f311b6-2ed5-455e-90ff-172afc6a22d5","Type":"ContainerDied","Data":"c45f70cf89a082188026e7427a482a152e4512172089a5f877ee239ef2bbe9fb"}
Sep 09 20:30:39.461892 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:39.461585    2578 scope.go:117] "RemoveContainer" containerID="38ac8e92cd06ec8aac3f00a799e4a9298df434f9e115499d18401ff549242788"
Sep 09 20:30:39.471362 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:39.471345    2578 scope.go:117] "RemoveContainer" containerID="3c89b26fff3bf2d19fc8cdb1b25888a317c04aa3d8afd77d6764a39965592588"
Sep 09 20:30:39.486048 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:39.486025    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"]
Sep 09 20:30:39.491522 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:39.491501    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-87c84fb8d-vwmc8"]
Sep 09 20:30:39.897282 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:39.897241    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" path="/var/lib/kubelet/pods/f6f311b6-2ed5-455e-90ff-172afc6a22d5/volumes"
Sep 09 20:30:40.674351 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:40.674292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:46.291665 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:46.291574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:46.392444 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:46.392405    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:46.844255 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:46.844212    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:46.856748 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:46.856716    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:30:47.509029 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:30:47.508996    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:31:28.271464 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:28.271427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:28.372815 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:28.372780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:33.033071 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.033037    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"]
Sep 09 20:31:33.033496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.033392    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" containerName="main"
Sep 09 20:31:33.033496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.033405    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" containerName="main"
Sep 09 20:31:33.033496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.033422    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" containerName="storage-initializer"
Sep 09 20:31:33.033496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.033428    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" containerName="storage-initializer"
Sep 09 20:31:33.033496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.033487    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6f311b6-2ed5-455e-90ff-172afc6a22d5" containerName="main"
Sep 09 20:31:33.036568 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.036544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.039245 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.039220    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:31:33.039374 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.039233    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 09 20:31:33.039936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.039909    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-n74bf\""
Sep 09 20:31:33.039936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.039927    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 09 20:31:33.040163 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.039995    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 09 20:31:33.040163 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.040034    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 09 20:31:33.045176 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.045154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-home\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.045293 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.045193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.045395 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.045281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.045457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.045390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.045457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.045437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.045567 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.045490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-model-cache\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.045567 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.045516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scb5l\" (UniqueName: \"kubernetes.io/projected/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kube-api-access-scb5l\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.045567 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.045544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-dshm\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.047408 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.047386    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"]
Sep 09 20:31:33.060894 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.060867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:33.146831 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.146787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-model-cache\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.146831 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.146833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scb5l\" (UniqueName: \"kubernetes.io/projected/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kube-api-access-scb5l\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.146831 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.146855    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-dshm\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147167 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.146877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-home\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147167 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.146910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147167 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.146938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147167 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.146980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147167 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.147016    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147472 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.147267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-model-cache\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147472 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.147382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-home\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147472 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.147429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147636 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.147479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.147766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.147745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.149178 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.149156    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-dshm\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.149444 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.149428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.159403 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.159383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scb5l\" (UniqueName: \"kubernetes.io/projected/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kube-api-access-scb5l\") pod \"tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.347922 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.347883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:33.379737 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.379706    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:33.437731 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.437704    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:33.625666 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.625631    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"]
Sep 09 20:31:33.627613 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:31:33.627585    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1561c5f7_de5e_4b5c_96c0_f1ac871be7e7.slice/crio-bab36a32328bc9611b310e8fd352da0fd96fd5d243196bd37053bfdcf9f40a14 WatchSource:0}: Error finding container bab36a32328bc9611b310e8fd352da0fd96fd5d243196bd37053bfdcf9f40a14: Status 404 returned error can't find the container with id bab36a32328bc9611b310e8fd352da0fd96fd5d243196bd37053bfdcf9f40a14
Sep 09 20:31:33.718719 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.718688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" event={"ID":"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7","Type":"ContainerStarted","Data":"63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a"}
Sep 09 20:31:33.718878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:33.718747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" event={"ID":"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7","Type":"ContainerStarted","Data":"bab36a32328bc9611b310e8fd352da0fd96fd5d243196bd37053bfdcf9f40a14"}
Sep 09 20:31:35.473273 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:35.473241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:36.735049 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:36.734968    2578 generic.go:356] "Generic (PLEG): container finished" podID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerID="63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a" exitCode=0
Sep 09 20:31:36.735049 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:36.735039    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" event={"ID":"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7","Type":"ContainerDied","Data":"63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a"}
Sep 09 20:31:37.747953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:37.747915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" event={"ID":"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7","Type":"ContainerStarted","Data":"ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d"}
Sep 09 20:31:37.767336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:37.767270    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" podStartSLOduration=2.433217312 podStartE2EDuration="4.767256818s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.334039506s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:37.765874779 +0000 UTC m=+2206.354528435" watchObservedRunningTime="2026-09-09 20:31:37.767256818 +0000 UTC m=+2206.355910521"
Sep 09 20:31:38.348344 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:38.348290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:41.025158 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:41.025102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:41.096526 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:41.096483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:43.348894 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:43.348854    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:43.361566 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:43.361543    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:43.785501 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:43.785421    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:31:46.905778 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:46.905737    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:51.580089 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:51.580051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:53.692779 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:53.692745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:53.797245 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:53.797211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:53.901048 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:53.901013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:54.843378 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:54.843296    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"]
Sep 09 20:31:54.856964 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:54.856923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:55.220252 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:55.219406    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"]
Sep 09 20:31:55.231740 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:55.231708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:56.210406 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:56.210376    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"]
Sep 09 20:31:56.236141 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:56.236106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:58.557185 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:58.557135    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:58.802999 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:31:58.802962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:09.864853 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:09.864810    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr" podUID="98678eff-e74f-49da-9808-a722fab2847d" containerName="main" containerID="cri-o://279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d" gracePeriod=45
Sep 09 20:32:10.107947 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.107921    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:32:10.242521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.242434    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g" podUID="837212d5-ea23-4ffa-8429-bd5914ce76f0" containerName="main" containerID="cri-o://d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b" gracePeriod=45
Sep 09 20:32:10.272684 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.272643    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-tmp-dir\") pod \"98678eff-e74f-49da-9808-a722fab2847d\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") "
Sep 09 20:32:10.272806 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.272708    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98678eff-e74f-49da-9808-a722fab2847d-tls-certs\") pod \"98678eff-e74f-49da-9808-a722fab2847d\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") "
Sep 09 20:32:10.272806 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.272755    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-model-cache\") pod \"98678eff-e74f-49da-9808-a722fab2847d\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") "
Sep 09 20:32:10.272806 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.272797    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-home\") pod \"98678eff-e74f-49da-9808-a722fab2847d\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") "
Sep 09 20:32:10.272976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.272821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-dshm\") pod \"98678eff-e74f-49da-9808-a722fab2847d\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") "
Sep 09 20:32:10.272976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.272841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njxr7\" (UniqueName: \"kubernetes.io/projected/98678eff-e74f-49da-9808-a722fab2847d-kube-api-access-njxr7\") pod \"98678eff-e74f-49da-9808-a722fab2847d\" (UID: \"98678eff-e74f-49da-9808-a722fab2847d\") "
Sep 09 20:32:10.272976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.272895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98678eff-e74f-49da-9808-a722fab2847d" (UID: "98678eff-e74f-49da-9808-a722fab2847d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.273121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.273069    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-home" (OuterVolumeSpecName: "home") pod "98678eff-e74f-49da-9808-a722fab2847d" (UID: "98678eff-e74f-49da-9808-a722fab2847d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.273121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.273092    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.273121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.273094    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-model-cache" (OuterVolumeSpecName: "model-cache") pod "98678eff-e74f-49da-9808-a722fab2847d" (UID: "98678eff-e74f-49da-9808-a722fab2847d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.274912 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.274886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98678eff-e74f-49da-9808-a722fab2847d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98678eff-e74f-49da-9808-a722fab2847d" (UID: "98678eff-e74f-49da-9808-a722fab2847d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:10.274970 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.274923    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-dshm" (OuterVolumeSpecName: "dshm") pod "98678eff-e74f-49da-9808-a722fab2847d" (UID: "98678eff-e74f-49da-9808-a722fab2847d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.274970 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.274929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98678eff-e74f-49da-9808-a722fab2847d-kube-api-access-njxr7" (OuterVolumeSpecName: "kube-api-access-njxr7") pod "98678eff-e74f-49da-9808-a722fab2847d" (UID: "98678eff-e74f-49da-9808-a722fab2847d"). InnerVolumeSpecName "kube-api-access-njxr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:10.374427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.374396    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.374427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.374423    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.374597 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.374433    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxr7\" (UniqueName: \"kubernetes.io/projected/98678eff-e74f-49da-9808-a722fab2847d-kube-api-access-njxr7\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.374597 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.374445    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98678eff-e74f-49da-9808-a722fab2847d-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.374597 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.374454    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98678eff-e74f-49da-9808-a722fab2847d-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.470321 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.470289    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:32:10.575660 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.575578    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-tmp-dir\") pod \"837212d5-ea23-4ffa-8429-bd5914ce76f0\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") "
Sep 09 20:32:10.575816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.575665    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-home\") pod \"837212d5-ea23-4ffa-8429-bd5914ce76f0\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") "
Sep 09 20:32:10.575816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.575706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndjz7\" (UniqueName: \"kubernetes.io/projected/837212d5-ea23-4ffa-8429-bd5914ce76f0-kube-api-access-ndjz7\") pod \"837212d5-ea23-4ffa-8429-bd5914ce76f0\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") "
Sep 09 20:32:10.575816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.575720    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-dshm\") pod \"837212d5-ea23-4ffa-8429-bd5914ce76f0\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") "
Sep 09 20:32:10.575816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.575738    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837212d5-ea23-4ffa-8429-bd5914ce76f0-tls-certs\") pod \"837212d5-ea23-4ffa-8429-bd5914ce76f0\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") "
Sep 09 20:32:10.575816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.575779    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-model-cache\") pod \"837212d5-ea23-4ffa-8429-bd5914ce76f0\" (UID: \"837212d5-ea23-4ffa-8429-bd5914ce76f0\") "
Sep 09 20:32:10.576078 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.575899    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-home" (OuterVolumeSpecName: "home") pod "837212d5-ea23-4ffa-8429-bd5914ce76f0" (UID: "837212d5-ea23-4ffa-8429-bd5914ce76f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.576078 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.575906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "837212d5-ea23-4ffa-8429-bd5914ce76f0" (UID: "837212d5-ea23-4ffa-8429-bd5914ce76f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.576078 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.576009    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.576078 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.576022    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.576238 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.576120    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "837212d5-ea23-4ffa-8429-bd5914ce76f0" (UID: "837212d5-ea23-4ffa-8429-bd5914ce76f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.577916 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.577895    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837212d5-ea23-4ffa-8429-bd5914ce76f0-kube-api-access-ndjz7" (OuterVolumeSpecName: "kube-api-access-ndjz7") pod "837212d5-ea23-4ffa-8429-bd5914ce76f0" (UID: "837212d5-ea23-4ffa-8429-bd5914ce76f0"). InnerVolumeSpecName "kube-api-access-ndjz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:10.577975 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.577927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-dshm" (OuterVolumeSpecName: "dshm") pod "837212d5-ea23-4ffa-8429-bd5914ce76f0" (UID: "837212d5-ea23-4ffa-8429-bd5914ce76f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:10.577975 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.577927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837212d5-ea23-4ffa-8429-bd5914ce76f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "837212d5-ea23-4ffa-8429-bd5914ce76f0" (UID: "837212d5-ea23-4ffa-8429-bd5914ce76f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:10.676817 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.676788    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.676817 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.676814    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjz7\" (UniqueName: \"kubernetes.io/projected/837212d5-ea23-4ffa-8429-bd5914ce76f0-kube-api-access-ndjz7\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.676817 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.676825    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/837212d5-ea23-4ffa-8429-bd5914ce76f0-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.677025 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.676834    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837212d5-ea23-4ffa-8429-bd5914ce76f0-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:10.899770 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.899734    2578 generic.go:356] "Generic (PLEG): container finished" podID="837212d5-ea23-4ffa-8429-bd5914ce76f0" containerID="d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b" exitCode=0
Sep 09 20:32:10.900207 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.899777    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"
Sep 09 20:32:10.900207 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.899817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g" event={"ID":"837212d5-ea23-4ffa-8429-bd5914ce76f0","Type":"ContainerDied","Data":"d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b"}
Sep 09 20:32:10.900207 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.899868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g" event={"ID":"837212d5-ea23-4ffa-8429-bd5914ce76f0","Type":"ContainerDied","Data":"d12374c95dbb524a7c258a06518291c3d67d0d4feaab46e7eebb7f4d54181d69"}
Sep 09 20:32:10.900207 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.899901    2578 scope.go:117] "RemoveContainer" containerID="d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b"
Sep 09 20:32:10.901354 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.901200    2578 generic.go:356] "Generic (PLEG): container finished" podID="98678eff-e74f-49da-9808-a722fab2847d" containerID="279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d" exitCode=0
Sep 09 20:32:10.901354 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.901244    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"
Sep 09 20:32:10.901354 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.901253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr" event={"ID":"98678eff-e74f-49da-9808-a722fab2847d","Type":"ContainerDied","Data":"279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d"}
Sep 09 20:32:10.901354 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.901293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr" event={"ID":"98678eff-e74f-49da-9808-a722fab2847d","Type":"ContainerDied","Data":"75110dee0c32f1b2dc33ea418bd309b0cf96be22f9901fc8badec0e80492d86e"}
Sep 09 20:32:10.909851 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.909831    2578 scope.go:117] "RemoveContainer" containerID="d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b"
Sep 09 20:32:10.910148 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:32:10.910129    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b\": container with ID starting with d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b not found: ID does not exist" containerID="d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b"
Sep 09 20:32:10.910227 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.910156    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b"} err="failed to get container status \"d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b\": rpc error: code = NotFound desc = could not find container \"d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b\": container with ID starting with d567dd22ca6837dd47079d62dc1d076a5b449a5627785e8c524a4be65325242b not found: ID does not exist"
Sep 09 20:32:10.910227 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.910172    2578 scope.go:117] "RemoveContainer" containerID="279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d"
Sep 09 20:32:10.918033 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.918014    2578 scope.go:117] "RemoveContainer" containerID="279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d"
Sep 09 20:32:10.918286 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:32:10.918266    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d\": container with ID starting with 279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d not found: ID does not exist" containerID="279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d"
Sep 09 20:32:10.918353 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.918293    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d"} err="failed to get container status \"279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d\": rpc error: code = NotFound desc = could not find container \"279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d\": container with ID starting with 279048132c6dcc88080281a9678143de20874e5a648c744015444dc29d9c674d not found: ID does not exist"
Sep 09 20:32:10.921954 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.921932    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"]
Sep 09 20:32:10.925949 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.925927    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g"]
Sep 09 20:32:10.935375 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.935352    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"]
Sep 09 20:32:10.938855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:10.938837    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-678pr"]
Sep 09 20:32:11.229694 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.229601    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n" podUID="f45c1efb-054c-4607-9c83-6c811cdfa960" containerName="main" containerID="cri-o://783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8" gracePeriod=45
Sep 09 20:32:11.455419 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.455379    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-9mm6g" podUID="837212d5-ea23-4ffa-8429-bd5914ce76f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:32:11.467121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.467097    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:32:11.584152 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.584054    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghzp5\" (UniqueName: \"kubernetes.io/projected/f45c1efb-054c-4607-9c83-6c811cdfa960-kube-api-access-ghzp5\") pod \"f45c1efb-054c-4607-9c83-6c811cdfa960\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") "
Sep 09 20:32:11.584152 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.584116    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f45c1efb-054c-4607-9c83-6c811cdfa960-tls-certs\") pod \"f45c1efb-054c-4607-9c83-6c811cdfa960\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") "
Sep 09 20:32:11.584446 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.584211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-dshm\") pod \"f45c1efb-054c-4607-9c83-6c811cdfa960\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") "
Sep 09 20:32:11.584446 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.584234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-model-cache\") pod \"f45c1efb-054c-4607-9c83-6c811cdfa960\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") "
Sep 09 20:32:11.584446 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.584262    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-tmp-dir\") pod \"f45c1efb-054c-4607-9c83-6c811cdfa960\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") "
Sep 09 20:32:11.584446 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.584287    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-home\") pod \"f45c1efb-054c-4607-9c83-6c811cdfa960\" (UID: \"f45c1efb-054c-4607-9c83-6c811cdfa960\") "
Sep 09 20:32:11.584661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.584559    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-model-cache" (OuterVolumeSpecName: "model-cache") pod "f45c1efb-054c-4607-9c83-6c811cdfa960" (UID: "f45c1efb-054c-4607-9c83-6c811cdfa960"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:11.584661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.584570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f45c1efb-054c-4607-9c83-6c811cdfa960" (UID: "f45c1efb-054c-4607-9c83-6c811cdfa960"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:11.584661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.584599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-home" (OuterVolumeSpecName: "home") pod "f45c1efb-054c-4607-9c83-6c811cdfa960" (UID: "f45c1efb-054c-4607-9c83-6c811cdfa960"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:11.586410 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.586383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45c1efb-054c-4607-9c83-6c811cdfa960-kube-api-access-ghzp5" (OuterVolumeSpecName: "kube-api-access-ghzp5") pod "f45c1efb-054c-4607-9c83-6c811cdfa960" (UID: "f45c1efb-054c-4607-9c83-6c811cdfa960"). InnerVolumeSpecName "kube-api-access-ghzp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:11.586524 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.586419    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-dshm" (OuterVolumeSpecName: "dshm") pod "f45c1efb-054c-4607-9c83-6c811cdfa960" (UID: "f45c1efb-054c-4607-9c83-6c811cdfa960"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:11.586524 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.586469    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45c1efb-054c-4607-9c83-6c811cdfa960-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f45c1efb-054c-4607-9c83-6c811cdfa960" (UID: "f45c1efb-054c-4607-9c83-6c811cdfa960"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:11.685453 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.685414    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:11.685453 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.685444    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:11.685661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.685478    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:11.685661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.685488    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f45c1efb-054c-4607-9c83-6c811cdfa960-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:11.685661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.685501    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzp5\" (UniqueName: \"kubernetes.io/projected/f45c1efb-054c-4607-9c83-6c811cdfa960-kube-api-access-ghzp5\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:11.685661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.685514    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f45c1efb-054c-4607-9c83-6c811cdfa960-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:11.896586 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.896551    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837212d5-ea23-4ffa-8429-bd5914ce76f0" path="/var/lib/kubelet/pods/837212d5-ea23-4ffa-8429-bd5914ce76f0/volumes"
Sep 09 20:32:11.897047 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.897026    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98678eff-e74f-49da-9808-a722fab2847d" path="/var/lib/kubelet/pods/98678eff-e74f-49da-9808-a722fab2847d/volumes"
Sep 09 20:32:11.908396 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.908370    2578 generic.go:356] "Generic (PLEG): container finished" podID="f45c1efb-054c-4607-9c83-6c811cdfa960" containerID="783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8" exitCode=0
Sep 09 20:32:11.908752 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.908413    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"
Sep 09 20:32:11.908752 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.908451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n" event={"ID":"f45c1efb-054c-4607-9c83-6c811cdfa960","Type":"ContainerDied","Data":"783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8"}
Sep 09 20:32:11.908752 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.908501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n" event={"ID":"f45c1efb-054c-4607-9c83-6c811cdfa960","Type":"ContainerDied","Data":"903f7f85df108733a854eef5940b61259f167d9361bf688c48c60421e67ae8b0"}
Sep 09 20:32:11.908752 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.908524    2578 scope.go:117] "RemoveContainer" containerID="783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8"
Sep 09 20:32:11.916986 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.916969    2578 scope.go:117] "RemoveContainer" containerID="783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8"
Sep 09 20:32:11.917246 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:32:11.917225    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8\": container with ID starting with 783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8 not found: ID does not exist" containerID="783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8"
Sep 09 20:32:11.917328 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.917254    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8"} err="failed to get container status \"783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8\": rpc error: code = NotFound desc = could not find container \"783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8\": container with ID starting with 783efa313e341480c3e4bebe96a337f6423ab276fa97722abfdffb310f16bed8 not found: ID does not exist"
Sep 09 20:32:11.923900 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.923870    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"]
Sep 09 20:32:11.927580 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:11.927552    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-dgt6n"]
Sep 09 20:32:13.896764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:13.896732    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f45c1efb-054c-4607-9c83-6c811cdfa960" path="/var/lib/kubelet/pods/f45c1efb-054c-4607-9c83-6c811cdfa960/volumes"
Sep 09 20:32:21.836145 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836104    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"]
Sep 09 20:32:21.836576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836472    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98678eff-e74f-49da-9808-a722fab2847d" containerName="main"
Sep 09 20:32:21.836576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836485    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="98678eff-e74f-49da-9808-a722fab2847d" containerName="main"
Sep 09 20:32:21.836576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836496    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45c1efb-054c-4607-9c83-6c811cdfa960" containerName="main"
Sep 09 20:32:21.836576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836502    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45c1efb-054c-4607-9c83-6c811cdfa960" containerName="main"
Sep 09 20:32:21.836576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836509    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837212d5-ea23-4ffa-8429-bd5914ce76f0" containerName="main"
Sep 09 20:32:21.836576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836515    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="837212d5-ea23-4ffa-8429-bd5914ce76f0" containerName="main"
Sep 09 20:32:21.836576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836563    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="837212d5-ea23-4ffa-8429-bd5914ce76f0" containerName="main"
Sep 09 20:32:21.836576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836569    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f45c1efb-054c-4607-9c83-6c811cdfa960" containerName="main"
Sep 09 20:32:21.836576 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.836578    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="98678eff-e74f-49da-9808-a722fab2847d" containerName="main"
Sep 09 20:32:21.841094 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.841076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:21.844567 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.844539    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 09 20:32:21.844567 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.844560    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 09 20:32:21.844760 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.844541    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 09 20:32:21.844760 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.844538    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-v8rc9\""
Sep 09 20:32:21.849152 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.849131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"]
Sep 09 20:32:21.922452 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.922408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:21.975475 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.975440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65gb\" (UniqueName: \"kubernetes.io/projected/d5849343-75ca-486e-ba5b-dac0fd7276ea-kube-api-access-k65gb\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:21.975668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.975496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-model-cache\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:21.975668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.975541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5849343-75ca-486e-ba5b-dac0fd7276ea-tls-certs\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:21.975668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.975606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-home\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:21.975668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.975640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:21.975833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:21.975683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-dshm\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.076702 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.076664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65gb\" (UniqueName: \"kubernetes.io/projected/d5849343-75ca-486e-ba5b-dac0fd7276ea-kube-api-access-k65gb\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.076886 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.076720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-model-cache\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.076886 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.076747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5849343-75ca-486e-ba5b-dac0fd7276ea-tls-certs\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.076886 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.076816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-home\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.076886 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.076874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.077149 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.076905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-dshm\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.077207 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.077170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-model-cache\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.077480 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.077458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-home\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.077554 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.077489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.079247 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.079229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-dshm\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.079675 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.079654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5849343-75ca-486e-ba5b-dac0fd7276ea-tls-certs\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.085775 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.085751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65gb\" (UniqueName: \"kubernetes.io/projected/d5849343-75ca-486e-ba5b-dac0fd7276ea-kube-api-access-k65gb\") pod \"late-v1-kserve-56cbc8456c-w6c2t\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.151579 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.151546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:22.424692 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.424663    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"]
Sep 09 20:32:22.426478 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:32:22.426439    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5849343_75ca_486e_ba5b_dac0fd7276ea.slice/crio-71c67316b3a62508b0bfda67fc5066a71171f789daf6640a2d32ddcfb6a3b6e3 WatchSource:0}: Error finding container 71c67316b3a62508b0bfda67fc5066a71171f789daf6640a2d32ddcfb6a3b6e3: Status 404 returned error can't find the container with id 71c67316b3a62508b0bfda67fc5066a71171f789daf6640a2d32ddcfb6a3b6e3
Sep 09 20:32:22.961374 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.961338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t" event={"ID":"d5849343-75ca-486e-ba5b-dac0fd7276ea","Type":"ContainerStarted","Data":"eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3"}
Sep 09 20:32:22.961784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.961390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t" event={"ID":"d5849343-75ca-486e-ba5b-dac0fd7276ea","Type":"ContainerStarted","Data":"71c67316b3a62508b0bfda67fc5066a71171f789daf6640a2d32ddcfb6a3b6e3"}
Sep 09 20:32:22.980193 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:22.980143    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t" podStartSLOduration=1.980126702 podStartE2EDuration="1.980126702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:22.977953911 +0000 UTC m=+2251.566607567" watchObservedRunningTime="2026-09-09 20:32:22.980126702 +0000 UTC m=+2251.568780360"
Sep 09 20:32:23.151879 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:23.151843    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:29.195571 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.195534    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"]
Sep 09 20:32:29.204656 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:32:29.204622    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:32:29.204656 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:32:29.204656 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:32:29.204656 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d" cmd=["/bin/sleep","15"]
Sep 09 20:32:29.204985 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.204679    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" message="" err=<
Sep 09 20:32:29.204985 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:32:29.204985 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:32:29.204985 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:32:29.204985 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:32:29.204695    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:32:29.204985 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:32:29.204985 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:32:29.204985 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" podUID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerName="main" containerID="cri-o://ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d"
Sep 09 20:32:29.204985 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.204732    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" podUID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerName="main" containerID="cri-o://ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d" gracePeriod=60
Sep 09 20:32:29.207642 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.207615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:29.279069 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.279025    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:29.775028 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.774982    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" podUID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": dial tcp 10.132.0.55:8000: connect: connection refused"
Sep 09 20:32:29.949709 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.949684    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:32:29.995487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.995448    2578 generic.go:356] "Generic (PLEG): container finished" podID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerID="ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d" exitCode=0
Sep 09 20:32:29.995648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.995502    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"
Sep 09 20:32:29.995648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.995550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" event={"ID":"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7","Type":"ContainerDied","Data":"ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d"}
Sep 09 20:32:29.995648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.995595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x" event={"ID":"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7","Type":"ContainerDied","Data":"bab36a32328bc9611b310e8fd352da0fd96fd5d243196bd37053bfdcf9f40a14"}
Sep 09 20:32:29.995648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:29.995617    2578 scope.go:117] "RemoveContainer" containerID="ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d"
Sep 09 20:32:30.004649 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.004624    2578 scope.go:117] "RemoveContainer" containerID="63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a"
Sep 09 20:32:30.012214 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.012192    2578 scope.go:117] "RemoveContainer" containerID="ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d"
Sep 09 20:32:30.012519 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:32:30.012495    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d\": container with ID starting with ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d not found: ID does not exist" containerID="ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d"
Sep 09 20:32:30.012570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.012527    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d"} err="failed to get container status \"ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d\": rpc error: code = NotFound desc = could not find container \"ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d\": container with ID starting with ce72c77b7e18cdb3eed0a5f2ed2a79a8af688b6cf64d46cb972f8f887c7d077d not found: ID does not exist"
Sep 09 20:32:30.012570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.012544    2578 scope.go:117] "RemoveContainer" containerID="63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a"
Sep 09 20:32:30.012802 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:32:30.012782    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a\": container with ID starting with 63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a not found: ID does not exist" containerID="63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a"
Sep 09 20:32:30.012846 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.012811    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a"} err="failed to get container status \"63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a\": rpc error: code = NotFound desc = could not find container \"63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a\": container with ID starting with 63b4c2ea2c679b285a01c9f94adfd777858987061b794da70c255814f824db7a not found: ID does not exist"
Sep 09 20:32:30.047187 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-home\") pod \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") "
Sep 09 20:32:30.047187 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047163    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tmp-dir\") pod \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") "
Sep 09 20:32:30.047187 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047183    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-model-cache\") pod \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") "
Sep 09 20:32:30.047359 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047227    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scb5l\" (UniqueName: \"kubernetes.io/projected/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kube-api-access-scb5l\") pod \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") "
Sep 09 20:32:30.047359 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047286    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kserve-provision-location\") pod \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") "
Sep 09 20:32:30.047359 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047346    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-dshm\") pod \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") "
Sep 09 20:32:30.047470 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-cabundle-cert\") pod \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") "
Sep 09 20:32:30.047470 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tls-certs\") pod \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\" (UID: \"1561c5f7-de5e-4b5c-96c0-f1ac871be7e7\") "
Sep 09 20:32:30.047470 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047449    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" (UID: "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:30.047470 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047455    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-home" (OuterVolumeSpecName: "home") pod "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" (UID: "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:30.047642 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047471    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" (UID: "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:30.047708 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047691    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:30.047772 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047717    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:30.047772 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047737    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:30.047899 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.047875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" (UID: "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:30.049579 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.049555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-dshm" (OuterVolumeSpecName: "dshm") pod "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" (UID: "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:30.049853 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.049829    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" (UID: "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:30.049853 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.049841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kube-api-access-scb5l" (OuterVolumeSpecName: "kube-api-access-scb5l") pod "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" (UID: "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7"). InnerVolumeSpecName "kube-api-access-scb5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:30.067416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.067394    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" (UID: "1561c5f7-de5e-4b5c-96c0-f1ac871be7e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:30.148783 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.148752    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scb5l\" (UniqueName: \"kubernetes.io/projected/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kube-api-access-scb5l\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:30.148783 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.148780    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:30.148966 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.148796    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:30.148966 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.148808    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:30.148966 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.148821    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:32:30.320265 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.320226    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"]
Sep 09 20:32:30.325141 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:30.325103    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-8589fdbb5-xpg2x"]
Sep 09 20:32:31.896183 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:31.896149    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" path="/var/lib/kubelet/pods/1561c5f7-de5e-4b5c-96c0-f1ac871be7e7/volumes"
Sep 09 20:32:32.152039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:32.151940    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:32.164451 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:32.164423    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:33.021497 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:33.021466    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:32:35.757749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:35.757713    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:41.336647 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:41.336591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:41.436227 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:41.436192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:51.332529 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.332489    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"]
Sep 09 20:32:51.334648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.332790    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerName="storage-initializer"
Sep 09 20:32:51.334648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.332801    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerName="storage-initializer"
Sep 09 20:32:51.334648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.332823    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerName="main"
Sep 09 20:32:51.334648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.332828    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerName="main"
Sep 09 20:32:51.334648 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.332878    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1561c5f7-de5e-4b5c-96c0-f1ac871be7e7" containerName="main"
Sep 09 20:32:51.335546 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.335528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.337991 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.337966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Sep 09 20:32:51.347696 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.347671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"]
Sep 09 20:32:51.420567 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.420540    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:51.428195 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.428170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-home\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.428346 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.428204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tls-certs\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.428346 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.428237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-model-cache\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.428442 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.428356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wplxk\" (UniqueName: \"kubernetes.io/projected/cafd46b1-3154-4735-b9d0-cde6f1ca4051-kube-api-access-wplxk\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.428442 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.428388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-dshm\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.428442 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.428405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tmp-dir\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.529123 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.529083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-home\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.529123 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.529128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tls-certs\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.529414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.529164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-model-cache\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.529414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.529196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wplxk\" (UniqueName: \"kubernetes.io/projected/cafd46b1-3154-4735-b9d0-cde6f1ca4051-kube-api-access-wplxk\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.529414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.529222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-dshm\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.529414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.529282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tmp-dir\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.529778 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.529580    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-home\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.529778 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.529632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-model-cache\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.529778 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.529657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tmp-dir\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.531679 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.531655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-dshm\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.531845 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.531831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tls-certs\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.537718 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.537695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wplxk\" (UniqueName: \"kubernetes.io/projected/cafd46b1-3154-4735-b9d0-cde6f1ca4051-kube-api-access-wplxk\") pod \"late-v2-kserve-86766546f-zntcd\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.645933 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.645890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:32:51.919531 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:51.919443    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"]
Sep 09 20:32:51.922810 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:32:51.922777    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafd46b1_3154_4735_b9d0_cde6f1ca4051.slice/crio-283f30e1c2a39b5b2c4004577bd194546e6f0fdd3e707e46f755441987a7f616 WatchSource:0}: Error finding container 283f30e1c2a39b5b2c4004577bd194546e6f0fdd3e707e46f755441987a7f616: Status 404 returned error can't find the container with id 283f30e1c2a39b5b2c4004577bd194546e6f0fdd3e707e46f755441987a7f616
Sep 09 20:32:52.104185 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:52.104146    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd" event={"ID":"cafd46b1-3154-4735-b9d0-cde6f1ca4051","Type":"ContainerStarted","Data":"f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f"}
Sep 09 20:32:52.104418 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:52.104221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd" event={"ID":"cafd46b1-3154-4735-b9d0-cde6f1ca4051","Type":"ContainerStarted","Data":"283f30e1c2a39b5b2c4004577bd194546e6f0fdd3e707e46f755441987a7f616"}
Sep 09 20:32:52.121024 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:52.120951    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd" podStartSLOduration=1.120927683 podStartE2EDuration="1.120927683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:52.120595461 +0000 UTC m=+2280.709249124" watchObservedRunningTime="2026-09-09 20:32:52.120927683 +0000 UTC m=+2280.709581342"
Sep 09 20:32:52.646417 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:32:52.646379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:33:01.646574 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:01.646532    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:33:01.659044 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:01.659014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:33:02.163547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:02.163512    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:33:15.719081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:15.719041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:15.817967 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:15.817932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:21.823033 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:21.822990    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:22.345827 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.345795    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"]
Sep 09 20:33:22.351131 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.351108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.353915 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.353893    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 09 20:33:22.354114 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.354092    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:33:22.354873 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.354857    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 09 20:33:22.355103 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.355083    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 09 20:33:22.355196 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.355186    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 09 20:33:22.355256 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.355214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-89pnb\""
Sep 09 20:33:22.367216 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.367195    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"]
Sep 09 20:33:22.393496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.393461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:22.397624 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.397593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-home\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.397751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.397648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-dshm\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.397751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.397689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eab91040-8645-4020-ab3d-1520d4fe3a68-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.397751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.397718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eab91040-8645-4020-ab3d-1520d4fe3a68-tls-certs\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.397885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.397753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.397885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.397805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.397885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.397824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzbw\" (UniqueName: \"kubernetes.io/projected/eab91040-8645-4020-ab3d-1520d4fe3a68-kube-api-access-2kzbw\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.397885 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.397880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-model-cache\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499239 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-model-cache\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-home\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-dshm\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eab91040-8645-4020-ab3d-1520d4fe3a68-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eab91040-8645-4020-ab3d-1520d4fe3a68-tls-certs\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499463 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499748 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499564    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499748 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzbw\" (UniqueName: \"kubernetes.io/projected/eab91040-8645-4020-ab3d-1520d4fe3a68-kube-api-access-2kzbw\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499748 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499660    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-model-cache\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499748 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-home\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499960 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.499960 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.499923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.500132 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.500111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eab91040-8645-4020-ab3d-1520d4fe3a68-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.501684 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.501665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-dshm\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.501975 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.501958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eab91040-8645-4020-ab3d-1520d4fe3a68-tls-certs\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.507574 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.507551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzbw\" (UniqueName: \"kubernetes.io/projected/eab91040-8645-4020-ab3d-1520d4fe3a68-kube-api-access-2kzbw\") pod \"tokenizer-template-args-test-kserve-5774447cc5-6nzz9\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.659715 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.659630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:22.661619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.661599    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:22.727467 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.727431    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:22.935693 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:22.935668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"]
Sep 09 20:33:22.937760 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:33:22.937738    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab91040_8645_4020_ab3d_1520d4fe3a68.slice/crio-6ede348944dd4906b2edef7276cfe24cf3e31b1de90a5c0b7729aa0f4693ab55 WatchSource:0}: Error finding container 6ede348944dd4906b2edef7276cfe24cf3e31b1de90a5c0b7729aa0f4693ab55: Status 404 returned error can't find the container with id 6ede348944dd4906b2edef7276cfe24cf3e31b1de90a5c0b7729aa0f4693ab55
Sep 09 20:33:23.246606 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:23.246509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" event={"ID":"eab91040-8645-4020-ab3d-1520d4fe3a68","Type":"ContainerStarted","Data":"2f885574eea67bfdd4ae64d211d4c8166860346b83ff649ea01224d02328d831"}
Sep 09 20:33:23.246606 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:23.246564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" event={"ID":"eab91040-8645-4020-ab3d-1520d4fe3a68","Type":"ContainerStarted","Data":"6ede348944dd4906b2edef7276cfe24cf3e31b1de90a5c0b7729aa0f4693ab55"}
Sep 09 20:33:26.264829 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:26.264793    2578 generic.go:356] "Generic (PLEG): container finished" podID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerID="2f885574eea67bfdd4ae64d211d4c8166860346b83ff649ea01224d02328d831" exitCode=0
Sep 09 20:33:26.265250 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:26.264872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" event={"ID":"eab91040-8645-4020-ab3d-1520d4fe3a68","Type":"ContainerDied","Data":"2f885574eea67bfdd4ae64d211d4c8166860346b83ff649ea01224d02328d831"}
Sep 09 20:33:27.272248 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:27.272211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" event={"ID":"eab91040-8645-4020-ab3d-1520d4fe3a68","Type":"ContainerStarted","Data":"458cc6bcb65318f81a0418a8e6e48694d64177dcf4c99bc4d3be2c2850be3545"}
Sep 09 20:33:27.295709 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:27.295654    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" podStartSLOduration=2.84402715 podStartE2EDuration="5.295639379s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.451612229s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:27.293884553 +0000 UTC m=+2315.882538235" watchObservedRunningTime="2026-09-09 20:33:27.295639379 +0000 UTC m=+2315.884293066"
Sep 09 20:33:27.487490 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:27.487460    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:27.587274 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:27.587183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:27.662082 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:27.662040    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:32.662686 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:32.662649    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:32.675150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:32.675124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:33.311355 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:33.311297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:33:35.418442 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:35.418409    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:38.421496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:38.421452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:38.585509 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:38.585471    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"]
Sep 09 20:33:38.609776 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:38.609740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:38.698750 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:38.698644    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:38.810347 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:38.810297    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"]
Sep 09 20:33:38.844394 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:38.844362    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:43.486213 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:43.486175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:48.423556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.423517    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"]
Sep 09 20:33:48.426066 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.426047    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.428899 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.428876    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 09 20:33:48.430043 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.430019    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 09 20:33:48.430150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.430022    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 09 20:33:48.430150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.430110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-4jkdg\""
Sep 09 20:33:48.437674 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.437651    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"]
Sep 09 20:33:48.439262 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.439241    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mtmf\" (UniqueName: \"kubernetes.io/projected/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-kube-api-access-7mtmf\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.439341 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.439271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-dshm\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.439389 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.439339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.439426 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.439384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-home\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.439426 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.439408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.439491 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.439437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.460353 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.460323    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:48.540290 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.540248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtmf\" (UniqueName: \"kubernetes.io/projected/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-kube-api-access-7mtmf\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.540290 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.540295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-dshm\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.540521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.540354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.540521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.540380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-home\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.540521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.540401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.540521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.540418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.540882 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.540857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.540882 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.540871    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-home\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.540997 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.540913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.542784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.542757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-dshm\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.543022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.543003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.548886 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.548866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtmf\" (UniqueName: \"kubernetes.io/projected/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-kube-api-access-7mtmf\") pod \"delw-v1-kserve-6cf8cbb847-c7rn8\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:48.736173 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:48.736075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:49.013334 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:49.011989    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"]
Sep 09 20:33:49.014992 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:33:49.014957    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cdd8eb_8434_44d5_ac1b_61d0cd12f82f.slice/crio-0fb34e0475ad109e3acd02474b3c174f2fdbaed55d2c035508b25840d8b95d8b WatchSource:0}: Error finding container 0fb34e0475ad109e3acd02474b3c174f2fdbaed55d2c035508b25840d8b95d8b: Status 404 returned error can't find the container with id 0fb34e0475ad109e3acd02474b3c174f2fdbaed55d2c035508b25840d8b95d8b
Sep 09 20:33:49.372396 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:49.372352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8" event={"ID":"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f","Type":"ContainerStarted","Data":"258b3b3ccfc9ddbb229be685f47594c18d93cd27920a1f4bade17bdcfa411bd3"}
Sep 09 20:33:49.372563 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:49.372416    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8" event={"ID":"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f","Type":"ContainerStarted","Data":"0fb34e0475ad109e3acd02474b3c174f2fdbaed55d2c035508b25840d8b95d8b"}
Sep 09 20:33:49.393703 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:49.393642    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8" podStartSLOduration=1.393622784 podStartE2EDuration="1.393622784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:49.391701928 +0000 UTC m=+2337.980355587" watchObservedRunningTime="2026-09-09 20:33:49.393622784 +0000 UTC m=+2337.982276440"
Sep 09 20:33:49.736962 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:49.736870    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:53.605477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.605432    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t" podUID="d5849343-75ca-486e-ba5b-dac0fd7276ea" containerName="main" containerID="cri-o://eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3" gracePeriod=45
Sep 09 20:33:53.832432 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.832388    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd" podUID="cafd46b1-3154-4735-b9d0-cde6f1ca4051" containerName="main" containerID="cri-o://f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f" gracePeriod=45
Sep 09 20:33:53.909963 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.909898    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:33:53.978413 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.978371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5849343-75ca-486e-ba5b-dac0fd7276ea-tls-certs\") pod \"d5849343-75ca-486e-ba5b-dac0fd7276ea\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") "
Sep 09 20:33:53.978413 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.978418    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-dshm\") pod \"d5849343-75ca-486e-ba5b-dac0fd7276ea\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") "
Sep 09 20:33:53.978667 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.978444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-model-cache\") pod \"d5849343-75ca-486e-ba5b-dac0fd7276ea\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") "
Sep 09 20:33:53.978667 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.978486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-tmp-dir\") pod \"d5849343-75ca-486e-ba5b-dac0fd7276ea\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") "
Sep 09 20:33:53.978667 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.978503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k65gb\" (UniqueName: \"kubernetes.io/projected/d5849343-75ca-486e-ba5b-dac0fd7276ea-kube-api-access-k65gb\") pod \"d5849343-75ca-486e-ba5b-dac0fd7276ea\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") "
Sep 09 20:33:53.978667 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.978521    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-home\") pod \"d5849343-75ca-486e-ba5b-dac0fd7276ea\" (UID: \"d5849343-75ca-486e-ba5b-dac0fd7276ea\") "
Sep 09 20:33:53.978888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.978793    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5849343-75ca-486e-ba5b-dac0fd7276ea" (UID: "d5849343-75ca-486e-ba5b-dac0fd7276ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:53.978949 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.978878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5849343-75ca-486e-ba5b-dac0fd7276ea" (UID: "d5849343-75ca-486e-ba5b-dac0fd7276ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:53.978949 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.978906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-home" (OuterVolumeSpecName: "home") pod "d5849343-75ca-486e-ba5b-dac0fd7276ea" (UID: "d5849343-75ca-486e-ba5b-dac0fd7276ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:53.980941 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.980913    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5849343-75ca-486e-ba5b-dac0fd7276ea-kube-api-access-k65gb" (OuterVolumeSpecName: "kube-api-access-k65gb") pod "d5849343-75ca-486e-ba5b-dac0fd7276ea" (UID: "d5849343-75ca-486e-ba5b-dac0fd7276ea"). InnerVolumeSpecName "kube-api-access-k65gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:53.981071 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.981053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-dshm" (OuterVolumeSpecName: "dshm") pod "d5849343-75ca-486e-ba5b-dac0fd7276ea" (UID: "d5849343-75ca-486e-ba5b-dac0fd7276ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:53.981128 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:53.981091    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5849343-75ca-486e-ba5b-dac0fd7276ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5849343-75ca-486e-ba5b-dac0fd7276ea" (UID: "d5849343-75ca-486e-ba5b-dac0fd7276ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:54.069603 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.069576    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:33:54.079586 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.079561    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tls-certs\") pod \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") "
Sep 09 20:33:54.079732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.079597    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-model-cache\") pod \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") "
Sep 09 20:33:54.079732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.079635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wplxk\" (UniqueName: \"kubernetes.io/projected/cafd46b1-3154-4735-b9d0-cde6f1ca4051-kube-api-access-wplxk\") pod \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") "
Sep 09 20:33:54.079732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.079668    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tmp-dir\") pod \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") "
Sep 09 20:33:54.079896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.079747    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-home\") pod \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") "
Sep 09 20:33:54.079896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.079795    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-dshm\") pod \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\" (UID: \"cafd46b1-3154-4735-b9d0-cde6f1ca4051\") "
Sep 09 20:33:54.079896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.079877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-model-cache" (OuterVolumeSpecName: "model-cache") pod "cafd46b1-3154-4735-b9d0-cde6f1ca4051" (UID: "cafd46b1-3154-4735-b9d0-cde6f1ca4051"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:54.080061 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.080013    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cafd46b1-3154-4735-b9d0-cde6f1ca4051" (UID: "cafd46b1-3154-4735-b9d0-cde6f1ca4051"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:54.080111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.080070    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.080111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.080079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-home" (OuterVolumeSpecName: "home") pod "cafd46b1-3154-4735-b9d0-cde6f1ca4051" (UID: "cafd46b1-3154-4735-b9d0-cde6f1ca4051"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:54.080111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.080091    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65gb\" (UniqueName: \"kubernetes.io/projected/d5849343-75ca-486e-ba5b-dac0fd7276ea-kube-api-access-k65gb\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.080111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.080103    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.080338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.080113    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5849343-75ca-486e-ba5b-dac0fd7276ea-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.080338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.080125    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.080338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.080135    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.080338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.080144    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5849343-75ca-486e-ba5b-dac0fd7276ea-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.081770 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.081745    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cafd46b1-3154-4735-b9d0-cde6f1ca4051" (UID: "cafd46b1-3154-4735-b9d0-cde6f1ca4051"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:54.081889 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.081780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafd46b1-3154-4735-b9d0-cde6f1ca4051-kube-api-access-wplxk" (OuterVolumeSpecName: "kube-api-access-wplxk") pod "cafd46b1-3154-4735-b9d0-cde6f1ca4051" (UID: "cafd46b1-3154-4735-b9d0-cde6f1ca4051"). InnerVolumeSpecName "kube-api-access-wplxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:54.081889 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.081784    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-dshm" (OuterVolumeSpecName: "dshm") pod "cafd46b1-3154-4735-b9d0-cde6f1ca4051" (UID: "cafd46b1-3154-4735-b9d0-cde6f1ca4051"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:54.180784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.180700    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wplxk\" (UniqueName: \"kubernetes.io/projected/cafd46b1-3154-4735-b9d0-cde6f1ca4051-kube-api-access-wplxk\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.180784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.180728    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.180784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.180738    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.180784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.180746    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cafd46b1-3154-4735-b9d0-cde6f1ca4051-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.180784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.180754    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cafd46b1-3154-4735-b9d0-cde6f1ca4051-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:33:54.396148 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.396109    2578 generic.go:356] "Generic (PLEG): container finished" podID="d5849343-75ca-486e-ba5b-dac0fd7276ea" containerID="eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3" exitCode=0
Sep 09 20:33:54.396148 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.396149    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"
Sep 09 20:33:54.396439 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.396195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t" event={"ID":"d5849343-75ca-486e-ba5b-dac0fd7276ea","Type":"ContainerDied","Data":"eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3"}
Sep 09 20:33:54.396439 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.396251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t" event={"ID":"d5849343-75ca-486e-ba5b-dac0fd7276ea","Type":"ContainerDied","Data":"71c67316b3a62508b0bfda67fc5066a71171f789daf6640a2d32ddcfb6a3b6e3"}
Sep 09 20:33:54.396439 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.396270    2578 scope.go:117] "RemoveContainer" containerID="eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3"
Sep 09 20:33:54.397610 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.397585    2578 generic.go:356] "Generic (PLEG): container finished" podID="cafd46b1-3154-4735-b9d0-cde6f1ca4051" containerID="f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f" exitCode=0
Sep 09 20:33:54.397752 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.397620    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"
Sep 09 20:33:54.397752 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.397621    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd" event={"ID":"cafd46b1-3154-4735-b9d0-cde6f1ca4051","Type":"ContainerDied","Data":"f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f"}
Sep 09 20:33:54.397876 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.397753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd" event={"ID":"cafd46b1-3154-4735-b9d0-cde6f1ca4051","Type":"ContainerDied","Data":"283f30e1c2a39b5b2c4004577bd194546e6f0fdd3e707e46f755441987a7f616"}
Sep 09 20:33:54.407512 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.407282    2578 scope.go:117] "RemoveContainer" containerID="eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3"
Sep 09 20:33:54.410213 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:33:54.410183    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3\": container with ID starting with eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3 not found: ID does not exist" containerID="eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3"
Sep 09 20:33:54.410340 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.410225    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3"} err="failed to get container status \"eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3\": rpc error: code = NotFound desc = could not find container \"eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3\": container with ID starting with eb237fff7e5d23a895522a8ab47bedeb699859724e6ec2f7614bb035b2ecbdf3 not found: ID does not exist"
Sep 09 20:33:54.410340 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.410248    2578 scope.go:117] "RemoveContainer" containerID="f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f"
Sep 09 20:33:54.418241 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.418226    2578 scope.go:117] "RemoveContainer" containerID="f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f"
Sep 09 20:33:54.418510 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:33:54.418492    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f\": container with ID starting with f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f not found: ID does not exist" containerID="f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f"
Sep 09 20:33:54.418560 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.418517    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f"} err="failed to get container status \"f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f\": rpc error: code = NotFound desc = could not find container \"f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f\": container with ID starting with f8b2ceab12c49b61b974fecb56306d6027b57c0825f4d0dde8ea55e39191dd2f not found: ID does not exist"
Sep 09 20:33:54.424577 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.424556    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"]
Sep 09 20:33:54.426821 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.426795    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-zntcd"]
Sep 09 20:33:54.436612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.436562    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"]
Sep 09 20:33:54.438811 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:54.438787    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-w6c2t"]
Sep 09 20:33:55.895919 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:55.895885    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cafd46b1-3154-4735-b9d0-cde6f1ca4051" path="/var/lib/kubelet/pods/cafd46b1-3154-4735-b9d0-cde6f1ca4051/volumes"
Sep 09 20:33:55.896323 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:55.896269    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5849343-75ca-486e-ba5b-dac0fd7276ea" path="/var/lib/kubelet/pods/d5849343-75ca-486e-ba5b-dac0fd7276ea/volumes"
Sep 09 20:33:57.048069 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.048032    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"]
Sep 09 20:33:57.048449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.048381    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5849343-75ca-486e-ba5b-dac0fd7276ea" containerName="main"
Sep 09 20:33:57.048449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.048393    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5849343-75ca-486e-ba5b-dac0fd7276ea" containerName="main"
Sep 09 20:33:57.048449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.048406    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafd46b1-3154-4735-b9d0-cde6f1ca4051" containerName="main"
Sep 09 20:33:57.048449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.048411    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafd46b1-3154-4735-b9d0-cde6f1ca4051" containerName="main"
Sep 09 20:33:57.048584 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.048460    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cafd46b1-3154-4735-b9d0-cde6f1ca4051" containerName="main"
Sep 09 20:33:57.048584 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.048470    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5849343-75ca-486e-ba5b-dac0fd7276ea" containerName="main"
Sep 09 20:33:57.051692 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.051672    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.055837 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.055818    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 09 20:33:57.070749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.070717    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"]
Sep 09 20:33:57.100909 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.100881    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:57.107040 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.107013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-home\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.107139 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.107063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.107139 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.107114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.107230 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.107155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.107230 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.107203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9rf2\" (UniqueName: \"kubernetes.io/projected/a9148041-7e00-43fc-aeaa-e628ee81f58c-kube-api-access-l9rf2\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.107230 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.107227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9148041-7e00-43fc-aeaa-e628ee81f58c-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.207705 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.207669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-home\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.207893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.207720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.207893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.207747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.207893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.207787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.207893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.207841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9rf2\" (UniqueName: \"kubernetes.io/projected/a9148041-7e00-43fc-aeaa-e628ee81f58c-kube-api-access-l9rf2\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.207893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.207873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9148041-7e00-43fc-aeaa-e628ee81f58c-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.208173 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.208111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-home\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.208173 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.208143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.208173 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.208171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.210126 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.210096    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.210536 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.210515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9148041-7e00-43fc-aeaa-e628ee81f58c-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.217622 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.217599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9rf2\" (UniqueName: \"kubernetes.io/projected/a9148041-7e00-43fc-aeaa-e628ee81f58c-kube-api-access-l9rf2\") pod \"delw-v2-kserve-5b84cc7ddf-cmk4g\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.361494 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.361460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:57.636827 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:57.636741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"]
Sep 09 20:33:57.639909 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:33:57.639868    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9148041_7e00_43fc_aeaa_e628ee81f58c.slice/crio-809d37024484ed34635bf821789930efb70e9830ffaca4367c7da7e69cfe7fc0 WatchSource:0}: Error finding container 809d37024484ed34635bf821789930efb70e9830ffaca4367c7da7e69cfe7fc0: Status 404 returned error can't find the container with id 809d37024484ed34635bf821789930efb70e9830ffaca4367c7da7e69cfe7fc0
Sep 09 20:33:58.421153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:58.421106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g" event={"ID":"a9148041-7e00-43fc-aeaa-e628ee81f58c","Type":"ContainerStarted","Data":"d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f"}
Sep 09 20:33:58.421578 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:58.421174    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g" event={"ID":"a9148041-7e00-43fc-aeaa-e628ee81f58c","Type":"ContainerStarted","Data":"809d37024484ed34635bf821789930efb70e9830ffaca4367c7da7e69cfe7fc0"}
Sep 09 20:33:58.737259 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:58.737163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:58.750025 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:58.750000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:33:58.781102 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:58.781047    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g" podStartSLOduration=1.781030324 podStartE2EDuration="1.781030324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:58.44722438 +0000 UTC m=+2347.035878036" watchObservedRunningTime="2026-09-09 20:33:58.781030324 +0000 UTC m=+2347.369684026"
Sep 09 20:33:59.361663 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:59.361627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:33:59.436878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:33:59.436849    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:34:07.362120 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:07.362084    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:34:07.374828 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:07.374799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:34:07.470891 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:07.470857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:34:12.863833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:12.863786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:12.887476 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:12.887432    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"]
Sep 09 20:34:12.896383 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:34:12.896345    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:34:12.896383 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:34:12.896383 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:34:12.896383 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="458cc6bcb65318f81a0418a8e6e48694d64177dcf4c99bc4d3be2c2850be3545" cmd=["/bin/sleep","15"]
Sep 09 20:34:12.896612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:12.896398    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" message="" err=<
Sep 09 20:34:12.896612 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:34:12.896612 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:34:12.896612 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:34:12.896612 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:34:12.896412    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:34:12.896612 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:34:12.896612 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:34:12.896612 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" podUID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerName="main" containerID="cri-o://458cc6bcb65318f81a0418a8e6e48694d64177dcf4c99bc4d3be2c2850be3545"
Sep 09 20:34:12.896612 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:12.896458    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" podUID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerName="main" containerID="cri-o://458cc6bcb65318f81a0418a8e6e48694d64177dcf4c99bc4d3be2c2850be3545" gracePeriod=60
Sep 09 20:34:12.978964 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:12.978933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:13.300723 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.300629    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" podUID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 09 20:34:13.491807 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.491771    2578 generic.go:356] "Generic (PLEG): container finished" podID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerID="458cc6bcb65318f81a0418a8e6e48694d64177dcf4c99bc4d3be2c2850be3545" exitCode=0
Sep 09 20:34:13.491973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.491834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" event={"ID":"eab91040-8645-4020-ab3d-1520d4fe3a68","Type":"ContainerDied","Data":"458cc6bcb65318f81a0418a8e6e48694d64177dcf4c99bc4d3be2c2850be3545"}
Sep 09 20:34:13.651810 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.651783    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:34:13.758767 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.758726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-model-cache\") pod \"eab91040-8645-4020-ab3d-1520d4fe3a68\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") "
Sep 09 20:34:13.758925 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.758774    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-kserve-provision-location\") pod \"eab91040-8645-4020-ab3d-1520d4fe3a68\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") "
Sep 09 20:34:13.758925 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.758804    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-home\") pod \"eab91040-8645-4020-ab3d-1520d4fe3a68\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") "
Sep 09 20:34:13.758925 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.758839    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kzbw\" (UniqueName: \"kubernetes.io/projected/eab91040-8645-4020-ab3d-1520d4fe3a68-kube-api-access-2kzbw\") pod \"eab91040-8645-4020-ab3d-1520d4fe3a68\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") "
Sep 09 20:34:13.758925 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.758858    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eab91040-8645-4020-ab3d-1520d4fe3a68-tls-certs\") pod \"eab91040-8645-4020-ab3d-1520d4fe3a68\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") "
Sep 09 20:34:13.759108 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.758932    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eab91040-8645-4020-ab3d-1520d4fe3a68-cabundle-cert\") pod \"eab91040-8645-4020-ab3d-1520d4fe3a68\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") "
Sep 09 20:34:13.759108 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.758984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-dshm\") pod \"eab91040-8645-4020-ab3d-1520d4fe3a68\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") "
Sep 09 20:34:13.759108 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.759011    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-tmp-dir\") pod \"eab91040-8645-4020-ab3d-1520d4fe3a68\" (UID: \"eab91040-8645-4020-ab3d-1520d4fe3a68\") "
Sep 09 20:34:13.759108 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.759083    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-model-cache" (OuterVolumeSpecName: "model-cache") pod "eab91040-8645-4020-ab3d-1520d4fe3a68" (UID: "eab91040-8645-4020-ab3d-1520d4fe3a68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:13.759342 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.759128    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-home" (OuterVolumeSpecName: "home") pod "eab91040-8645-4020-ab3d-1520d4fe3a68" (UID: "eab91040-8645-4020-ab3d-1520d4fe3a68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:13.759412 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.759374    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:13.759461 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.759411    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:13.759461 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.759391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eab91040-8645-4020-ab3d-1520d4fe3a68" (UID: "eab91040-8645-4020-ab3d-1520d4fe3a68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:13.759461 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.759402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab91040-8645-4020-ab3d-1520d4fe3a68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eab91040-8645-4020-ab3d-1520d4fe3a68" (UID: "eab91040-8645-4020-ab3d-1520d4fe3a68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:13.761244 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.761196    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-dshm" (OuterVolumeSpecName: "dshm") pod "eab91040-8645-4020-ab3d-1520d4fe3a68" (UID: "eab91040-8645-4020-ab3d-1520d4fe3a68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:13.761430 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.761251    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eab91040-8645-4020-ab3d-1520d4fe3a68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eab91040-8645-4020-ab3d-1520d4fe3a68" (UID: "eab91040-8645-4020-ab3d-1520d4fe3a68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:13.761430 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.761336    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab91040-8645-4020-ab3d-1520d4fe3a68-kube-api-access-2kzbw" (OuterVolumeSpecName: "kube-api-access-2kzbw") pod "eab91040-8645-4020-ab3d-1520d4fe3a68" (UID: "eab91040-8645-4020-ab3d-1520d4fe3a68"). InnerVolumeSpecName "kube-api-access-2kzbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:13.779193 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.779167    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eab91040-8645-4020-ab3d-1520d4fe3a68" (UID: "eab91040-8645-4020-ab3d-1520d4fe3a68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:13.859792 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.859764    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:13.859792 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.859792    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:13.859980 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.859804    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eab91040-8645-4020-ab3d-1520d4fe3a68-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:13.859980 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.859815    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kzbw\" (UniqueName: \"kubernetes.io/projected/eab91040-8645-4020-ab3d-1520d4fe3a68-kube-api-access-2kzbw\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:13.859980 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.859824    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eab91040-8645-4020-ab3d-1520d4fe3a68-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:13.859980 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:13.859832    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eab91040-8645-4020-ab3d-1520d4fe3a68-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:14.497819 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:14.497784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9" event={"ID":"eab91040-8645-4020-ab3d-1520d4fe3a68","Type":"ContainerDied","Data":"6ede348944dd4906b2edef7276cfe24cf3e31b1de90a5c0b7729aa0f4693ab55"}
Sep 09 20:34:14.498258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:14.497843    2578 scope.go:117] "RemoveContainer" containerID="458cc6bcb65318f81a0418a8e6e48694d64177dcf4c99bc4d3be2c2850be3545"
Sep 09 20:34:14.498258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:14.497800    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"
Sep 09 20:34:14.507023 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:14.507005    2578 scope.go:117] "RemoveContainer" containerID="2f885574eea67bfdd4ae64d211d4c8166860346b83ff649ea01224d02328d831"
Sep 09 20:34:14.523332 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:14.523293    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"]
Sep 09 20:34:14.531887 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:14.531862    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5774447cc5-6nzz9"]
Sep 09 20:34:15.899651 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:15.899607    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eab91040-8645-4020-ab3d-1520d4fe3a68" path="/var/lib/kubelet/pods/eab91040-8645-4020-ab3d-1520d4fe3a68/volumes"
Sep 09 20:34:22.133453 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:22.133405    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:28.050935 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:28.050895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:28.140723 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:28.140685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:30.683233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:30.683192    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"]
Sep 09 20:34:30.707249 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:30.707207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:32.630716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:32.630683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:40.198909 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:40.198873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:45.710751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:45.710703    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g" podUID="a9148041-7e00-43fc-aeaa-e628ee81f58c" containerName="main" containerID="cri-o://d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f" gracePeriod=45
Sep 09 20:34:45.951618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:45.951592    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:34:46.048619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.048533    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-model-cache\") pod \"a9148041-7e00-43fc-aeaa-e628ee81f58c\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") "
Sep 09 20:34:46.048619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.048577    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-dshm\") pod \"a9148041-7e00-43fc-aeaa-e628ee81f58c\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") "
Sep 09 20:34:46.048619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.048620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-tmp-dir\") pod \"a9148041-7e00-43fc-aeaa-e628ee81f58c\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") "
Sep 09 20:34:46.048906 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.048639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9148041-7e00-43fc-aeaa-e628ee81f58c-tls-certs\") pod \"a9148041-7e00-43fc-aeaa-e628ee81f58c\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") "
Sep 09 20:34:46.048906 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.048658    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-home\") pod \"a9148041-7e00-43fc-aeaa-e628ee81f58c\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") "
Sep 09 20:34:46.048906 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.048674    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9rf2\" (UniqueName: \"kubernetes.io/projected/a9148041-7e00-43fc-aeaa-e628ee81f58c-kube-api-access-l9rf2\") pod \"a9148041-7e00-43fc-aeaa-e628ee81f58c\" (UID: \"a9148041-7e00-43fc-aeaa-e628ee81f58c\") "
Sep 09 20:34:46.048906 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.048851    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9148041-7e00-43fc-aeaa-e628ee81f58c" (UID: "a9148041-7e00-43fc-aeaa-e628ee81f58c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:46.048906 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.048859    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9148041-7e00-43fc-aeaa-e628ee81f58c" (UID: "a9148041-7e00-43fc-aeaa-e628ee81f58c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:46.049118 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.048997    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-home" (OuterVolumeSpecName: "home") pod "a9148041-7e00-43fc-aeaa-e628ee81f58c" (UID: "a9148041-7e00-43fc-aeaa-e628ee81f58c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:46.051011 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.050988    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-dshm" (OuterVolumeSpecName: "dshm") pod "a9148041-7e00-43fc-aeaa-e628ee81f58c" (UID: "a9148041-7e00-43fc-aeaa-e628ee81f58c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:46.051103 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.051085    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9148041-7e00-43fc-aeaa-e628ee81f58c-kube-api-access-l9rf2" (OuterVolumeSpecName: "kube-api-access-l9rf2") pod "a9148041-7e00-43fc-aeaa-e628ee81f58c" (UID: "a9148041-7e00-43fc-aeaa-e628ee81f58c"). InnerVolumeSpecName "kube-api-access-l9rf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:46.051154 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.051122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9148041-7e00-43fc-aeaa-e628ee81f58c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9148041-7e00-43fc-aeaa-e628ee81f58c" (UID: "a9148041-7e00-43fc-aeaa-e628ee81f58c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:46.149654 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.149620    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:46.149654 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.149648    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9148041-7e00-43fc-aeaa-e628ee81f58c-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:46.149654 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.149657    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:46.149869 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.149667    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9rf2\" (UniqueName: \"kubernetes.io/projected/a9148041-7e00-43fc-aeaa-e628ee81f58c-kube-api-access-l9rf2\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:46.149869 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.149677    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:46.149869 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.149685    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9148041-7e00-43fc-aeaa-e628ee81f58c-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:34:46.645290 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.645250    2578 generic.go:356] "Generic (PLEG): container finished" podID="a9148041-7e00-43fc-aeaa-e628ee81f58c" containerID="d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f" exitCode=0
Sep 09 20:34:46.645496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.645296    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"
Sep 09 20:34:46.645496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.645293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g" event={"ID":"a9148041-7e00-43fc-aeaa-e628ee81f58c","Type":"ContainerDied","Data":"d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f"}
Sep 09 20:34:46.645496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.645443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g" event={"ID":"a9148041-7e00-43fc-aeaa-e628ee81f58c","Type":"ContainerDied","Data":"809d37024484ed34635bf821789930efb70e9830ffaca4367c7da7e69cfe7fc0"}
Sep 09 20:34:46.645496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.645478    2578 scope.go:117] "RemoveContainer" containerID="d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f"
Sep 09 20:34:46.654073 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.654053    2578 scope.go:117] "RemoveContainer" containerID="d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f"
Sep 09 20:34:46.654352 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:34:46.654325    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f\": container with ID starting with d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f not found: ID does not exist" containerID="d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f"
Sep 09 20:34:46.654452 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.654364    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f"} err="failed to get container status \"d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f\": rpc error: code = NotFound desc = could not find container \"d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f\": container with ID starting with d0458bb79a6bf3e1bf75945d211799bc0de310025c3be0137bf8b14b9645fc0f not found: ID does not exist"
Sep 09 20:34:46.671631 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.671604    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"]
Sep 09 20:34:46.678059 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:46.678036    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-cmk4g"]
Sep 09 20:34:47.308845 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:47.308812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:47.534229 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:47.534196    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"]
Sep 09 20:34:47.617870 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:47.617837    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:47.896062 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:47.895984    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9148041-7e00-43fc-aeaa-e628ee81f58c" path="/var/lib/kubelet/pods/a9148041-7e00-43fc-aeaa-e628ee81f58c/volumes"
Sep 09 20:34:52.023632 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:52.023604    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:34:52.029050 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:52.029030    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:34:52.336834 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:52.336749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:55.131527 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.131494    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"]
Sep 09 20:34:55.132150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.131952    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerName="storage-initializer"
Sep 09 20:34:55.132150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.131973    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerName="storage-initializer"
Sep 09 20:34:55.132150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.131991    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerName="main"
Sep 09 20:34:55.132150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.132000    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerName="main"
Sep 09 20:34:55.132150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.132023    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9148041-7e00-43fc-aeaa-e628ee81f58c" containerName="main"
Sep 09 20:34:55.132150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.132032    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9148041-7e00-43fc-aeaa-e628ee81f58c" containerName="main"
Sep 09 20:34:55.132150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.132120    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eab91040-8645-4020-ab3d-1520d4fe3a68" containerName="main"
Sep 09 20:34:55.132150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.132133    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9148041-7e00-43fc-aeaa-e628ee81f58c" containerName="main"
Sep 09 20:34:55.134930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.134909    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.137646 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.137626    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 09 20:34:55.138706 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.138686    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 09 20:34:55.138804 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.138710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-kg8l5\""
Sep 09 20:34:55.138804 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.138715    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 09 20:34:55.145045 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.145025    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"]
Sep 09 20:34:55.158586 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.158556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:55.230424 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.230396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.230559 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.230426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s295\" (UniqueName: \"kubernetes.io/projected/733c3c2b-b028-49a0-99f1-0fa2a475a16a-kube-api-access-8s295\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.230559 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.230463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-home\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.230647 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.230566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.230647 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.230599    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-dshm\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.230647 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.230640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.331344 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.331288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.331521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.331351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-dshm\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.331521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.331390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.331521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.331433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.331669 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.331547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s295\" (UniqueName: \"kubernetes.io/projected/733c3c2b-b028-49a0-99f1-0fa2a475a16a-kube-api-access-8s295\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.331669 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.331596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-home\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.331855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.331831    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.331927 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.331891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.331991 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.331973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-home\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.333609 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.333583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-dshm\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.333894 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.333874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.339865 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.339846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s295\" (UniqueName: \"kubernetes.io/projected/733c3c2b-b028-49a0-99f1-0fa2a475a16a-kube-api-access-8s295\") pod \"roll-v1-kserve-69cf8bcfc-sg76l\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.445397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.445302    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:34:55.731101 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:55.731001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"]
Sep 09 20:34:55.735905 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:34:55.735875    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733c3c2b_b028_49a0_99f1_0fa2a475a16a.slice/crio-337d258e2d56a0d760e0e993b1c4fc9a9457a2f1d0ad40adad6636eb750caea0 WatchSource:0}: Error finding container 337d258e2d56a0d760e0e993b1c4fc9a9457a2f1d0ad40adad6636eb750caea0: Status 404 returned error can't find the container with id 337d258e2d56a0d760e0e993b1c4fc9a9457a2f1d0ad40adad6636eb750caea0
Sep 09 20:34:56.690677 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:56.690643    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l" event={"ID":"733c3c2b-b028-49a0-99f1-0fa2a475a16a","Type":"ContainerStarted","Data":"287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20"}
Sep 09 20:34:56.691063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:56.690694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l" event={"ID":"733c3c2b-b028-49a0-99f1-0fa2a475a16a","Type":"ContainerStarted","Data":"337d258e2d56a0d760e0e993b1c4fc9a9457a2f1d0ad40adad6636eb750caea0"}
Sep 09 20:34:56.712345 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:56.712255    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l" podStartSLOduration=1.712241175 podStartE2EDuration="1.712241175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:56.710285891 +0000 UTC m=+2405.298939548" watchObservedRunningTime="2026-09-09 20:34:56.712241175 +0000 UTC m=+2405.300894831"
Sep 09 20:34:57.446448 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:34:57.446414    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:35:00.971893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:00.971859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:01.065789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:01.065753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:02.553450 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.553402    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8" podUID="b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" containerName="main" containerID="cri-o://258b3b3ccfc9ddbb229be685f47594c18d93cd27920a1f4bade17bdcfa411bd3" gracePeriod=45
Sep 09 20:35:02.721501 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.721470    2578 generic.go:356] "Generic (PLEG): container finished" podID="b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" containerID="258b3b3ccfc9ddbb229be685f47594c18d93cd27920a1f4bade17bdcfa411bd3" exitCode=0
Sep 09 20:35:02.721659 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.721518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8" event={"ID":"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f","Type":"ContainerDied","Data":"258b3b3ccfc9ddbb229be685f47594c18d93cd27920a1f4bade17bdcfa411bd3"}
Sep 09 20:35:02.802827 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.802801    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:35:02.895255 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895227    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tls-certs\") pod \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") "
Sep 09 20:35:02.895441 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-model-cache\") pod \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") "
Sep 09 20:35:02.895441 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tmp-dir\") pod \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") "
Sep 09 20:35:02.895441 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895341    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-home\") pod \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") "
Sep 09 20:35:02.895441 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mtmf\" (UniqueName: \"kubernetes.io/projected/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-kube-api-access-7mtmf\") pod \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") "
Sep 09 20:35:02.895652 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895477    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-dshm\") pod \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\" (UID: \"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f\") "
Sep 09 20:35:02.895652 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895548    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" (UID: "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:02.895652 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895625    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-home" (OuterVolumeSpecName: "home") pod "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" (UID: "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:02.895652 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" (UID: "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:02.895878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895744    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:35:02.895878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895764    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:35:02.895878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.895779    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:35:02.897558 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.897528    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" (UID: "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:02.897558 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.897534    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-dshm" (OuterVolumeSpecName: "dshm") pod "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" (UID: "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:02.897700 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.897612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-kube-api-access-7mtmf" (OuterVolumeSpecName: "kube-api-access-7mtmf") pod "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" (UID: "b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f"). InnerVolumeSpecName "kube-api-access-7mtmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:02.996867 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.996833    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:35:02.996867 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.996856    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:35:02.996867 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:02.996866    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mtmf\" (UniqueName: \"kubernetes.io/projected/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f-kube-api-access-7mtmf\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:35:03.732850 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:03.732808    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8" event={"ID":"b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f","Type":"ContainerDied","Data":"0fb34e0475ad109e3acd02474b3c174f2fdbaed55d2c035508b25840d8b95d8b"}
Sep 09 20:35:03.733233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:03.732824    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"
Sep 09 20:35:03.733233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:03.732913    2578 scope.go:117] "RemoveContainer" containerID="258b3b3ccfc9ddbb229be685f47594c18d93cd27920a1f4bade17bdcfa411bd3"
Sep 09 20:35:03.759053 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:03.759027    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"]
Sep 09 20:35:03.764105 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:03.764076    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-c7rn8"]
Sep 09 20:35:03.895734 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:03.895701    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" path="/var/lib/kubelet/pods/b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f/volumes"
Sep 09 20:35:05.446238 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:05.446191    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:35:05.458666 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:05.458640    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:35:05.664012 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:05.663977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:05.753365 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:05.753257    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:35:07.735529 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:07.735492    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:13.965347 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:13.965288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:14.045050 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:14.044977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:29.339266 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:29.339225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:31.671159 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.671124    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"]
Sep 09 20:35:31.671583 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.671511    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" containerName="main"
Sep 09 20:35:31.671583 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.671525    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" containerName="main"
Sep 09 20:35:31.671673 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.671610    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0cdd8eb-8434-44d5-ac1b-61d0cd12f82f" containerName="main"
Sep 09 20:35:31.673483 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.673466    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.677616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.677590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-n5lvh\""
Sep 09 20:35:31.677616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.677606    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 09 20:35:31.677803 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.677643    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 09 20:35:31.677803 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.677605    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 09 20:35:31.697702 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.697670    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"]
Sep 09 20:35:31.740887 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.740849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.740887 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.740888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.741111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.740906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.741111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.740976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.741111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.741033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a64a19-6fa3-4acf-9763-0d0558f95c74-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.741111 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.741051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwgl\" (UniqueName: \"kubernetes.io/projected/17a64a19-6fa3-4acf-9763-0d0558f95c74-kube-api-access-bdwgl\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.746644 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.746617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:31.841639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.841602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.841813 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.841698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a64a19-6fa3-4acf-9763-0d0558f95c74-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.841813 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.841726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwgl\" (UniqueName: \"kubernetes.io/projected/17a64a19-6fa3-4acf-9763-0d0558f95c74-kube-api-access-bdwgl\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.841813 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.841784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.841970 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.841820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.841970 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.841849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.842078 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.842017    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.842203 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.842182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.842258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.842222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.844128 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.844103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.844836 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.844812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a64a19-6fa3-4acf-9763-0d0558f95c74-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.852477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.852444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwgl\" (UniqueName: \"kubernetes.io/projected/17a64a19-6fa3-4acf-9763-0d0558f95c74-kube-api-access-bdwgl\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:31.983746 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:31.983660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:32.020488 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:32.020449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:32.254987 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:32.254894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"]
Sep 09 20:35:32.261341 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:35:32.261286    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a64a19_6fa3_4acf_9763_0d0558f95c74.slice/crio-eaada3cddbd8c26fef8b71d75814c9f7539228847ed611d3bc6b860c08a73ac6 WatchSource:0}: Error finding container eaada3cddbd8c26fef8b71d75814c9f7539228847ed611d3bc6b860c08a73ac6: Status 404 returned error can't find the container with id eaada3cddbd8c26fef8b71d75814c9f7539228847ed611d3bc6b860c08a73ac6
Sep 09 20:35:32.862107 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:32.862072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf" event={"ID":"17a64a19-6fa3-4acf-9763-0d0558f95c74","Type":"ContainerStarted","Data":"c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110"}
Sep 09 20:35:32.862580 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:32.862123    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf" event={"ID":"17a64a19-6fa3-4acf-9763-0d0558f95c74","Type":"ContainerStarted","Data":"eaada3cddbd8c26fef8b71d75814c9f7539228847ed611d3bc6b860c08a73ac6"}
Sep 09 20:35:32.884297 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:32.884246    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf" podStartSLOduration=1.884231489 podStartE2EDuration="1.884231489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:32.882387052 +0000 UTC m=+2441.471040710" watchObservedRunningTime="2026-09-09 20:35:32.884231489 +0000 UTC m=+2441.472885179"
Sep 09 20:35:32.984690 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:32.984656    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:35.423427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:35.423388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:35.494947 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:35.494909    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:41.984372 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:41.984334    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:41.996917 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:41.996895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:42.918745 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:42.918706    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:35:52.453435 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:35:52.453395    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:04.540443 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:04.540397    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:09.146573 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.146522    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:09.157088 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.157056    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"]
Sep 09 20:36:09.166486 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:36:09.166449    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:36:09.166486 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:36:09.166486 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:36:09.166486 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110" cmd=["/bin/sleep","15"]
Sep 09 20:36:09.166749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.166503    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf" message="" err=<
Sep 09 20:36:09.166749 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:36:09.166749 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:36:09.166749 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:36:09.166749 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:36:09.166517    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:36:09.166749 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:36:09.166749 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:36:09.166749 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf" podUID="17a64a19-6fa3-4acf-9763-0d0558f95c74" containerName="main" containerID="cri-o://c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110"
Sep 09 20:36:09.166749 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.166554    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf" podUID="17a64a19-6fa3-4acf-9763-0d0558f95c74" containerName="main" containerID="cri-o://c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110" gracePeriod=60
Sep 09 20:36:09.245567 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.245530    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:09.512764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.512730    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:36:09.574908 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.574877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-model-cache\") pod \"17a64a19-6fa3-4acf-9763-0d0558f95c74\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") "
Sep 09 20:36:09.575063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.574958    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-home\") pod \"17a64a19-6fa3-4acf-9763-0d0558f95c74\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") "
Sep 09 20:36:09.575063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.574976    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-dshm\") pod \"17a64a19-6fa3-4acf-9763-0d0558f95c74\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") "
Sep 09 20:36:09.575063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.575003    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-tmp-dir\") pod \"17a64a19-6fa3-4acf-9763-0d0558f95c74\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") "
Sep 09 20:36:09.575063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.575040    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a64a19-6fa3-4acf-9763-0d0558f95c74-tls-certs\") pod \"17a64a19-6fa3-4acf-9763-0d0558f95c74\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") "
Sep 09 20:36:09.575063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.575059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdwgl\" (UniqueName: \"kubernetes.io/projected/17a64a19-6fa3-4acf-9763-0d0558f95c74-kube-api-access-bdwgl\") pod \"17a64a19-6fa3-4acf-9763-0d0558f95c74\" (UID: \"17a64a19-6fa3-4acf-9763-0d0558f95c74\") "
Sep 09 20:36:09.575367 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.575189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-model-cache" (OuterVolumeSpecName: "model-cache") pod "17a64a19-6fa3-4acf-9763-0d0558f95c74" (UID: "17a64a19-6fa3-4acf-9763-0d0558f95c74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:09.575367 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.575226    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-home" (OuterVolumeSpecName: "home") pod "17a64a19-6fa3-4acf-9763-0d0558f95c74" (UID: "17a64a19-6fa3-4acf-9763-0d0558f95c74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:09.575367 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.575291    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17a64a19-6fa3-4acf-9763-0d0558f95c74" (UID: "17a64a19-6fa3-4acf-9763-0d0558f95c74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:09.575367 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.575346    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:36:09.575367 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.575359    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:36:09.577397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.577365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-dshm" (OuterVolumeSpecName: "dshm") pod "17a64a19-6fa3-4acf-9763-0d0558f95c74" (UID: "17a64a19-6fa3-4acf-9763-0d0558f95c74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:09.577397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.577379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a64a19-6fa3-4acf-9763-0d0558f95c74-kube-api-access-bdwgl" (OuterVolumeSpecName: "kube-api-access-bdwgl") pod "17a64a19-6fa3-4acf-9763-0d0558f95c74" (UID: "17a64a19-6fa3-4acf-9763-0d0558f95c74"). InnerVolumeSpecName "kube-api-access-bdwgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:09.577588 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.577510    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a64a19-6fa3-4acf-9763-0d0558f95c74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17a64a19-6fa3-4acf-9763-0d0558f95c74" (UID: "17a64a19-6fa3-4acf-9763-0d0558f95c74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:09.676075 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.676035    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdwgl\" (UniqueName: \"kubernetes.io/projected/17a64a19-6fa3-4acf-9763-0d0558f95c74-kube-api-access-bdwgl\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:36:09.676075 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.676069    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:36:09.676075 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.676083    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17a64a19-6fa3-4acf-9763-0d0558f95c74-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:36:09.676341 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:09.676095    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17a64a19-6fa3-4acf-9763-0d0558f95c74-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:36:10.035365 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:10.035330    2578 generic.go:356] "Generic (PLEG): container finished" podID="17a64a19-6fa3-4acf-9763-0d0558f95c74" containerID="c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110" exitCode=0
Sep 09 20:36:10.035365 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:10.035369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf" event={"ID":"17a64a19-6fa3-4acf-9763-0d0558f95c74","Type":"ContainerDied","Data":"c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110"}
Sep 09 20:36:10.035619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:10.035432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf" event={"ID":"17a64a19-6fa3-4acf-9763-0d0558f95c74","Type":"ContainerDied","Data":"eaada3cddbd8c26fef8b71d75814c9f7539228847ed611d3bc6b860c08a73ac6"}
Sep 09 20:36:10.035619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:10.035447    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"
Sep 09 20:36:10.035619 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:10.035455    2578 scope.go:117] "RemoveContainer" containerID="c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110"
Sep 09 20:36:10.044269 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:10.044250    2578 scope.go:117] "RemoveContainer" containerID="c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110"
Sep 09 20:36:10.044582 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:36:10.044561    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110\": container with ID starting with c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110 not found: ID does not exist" containerID="c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110"
Sep 09 20:36:10.044639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:10.044591    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110"} err="failed to get container status \"c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110\": rpc error: code = NotFound desc = could not find container \"c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110\": container with ID starting with c61ec67faf146cfff1ce61538b0d76530cbda7f5121ed805b977d3683035a110 not found: ID does not exist"
Sep 09 20:36:10.053983 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:10.053960    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"]
Sep 09 20:36:10.058146 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:10.058120    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-2qswf"]
Sep 09 20:36:11.895914 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:11.895878    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a64a19-6fa3-4acf-9763-0d0558f95c74" path="/var/lib/kubelet/pods/17a64a19-6fa3-4acf-9763-0d0558f95c74/volumes"
Sep 09 20:36:45.214301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:45.214221    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:45.378858 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:45.378818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:45.478873 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:45.478796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:45.543759 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:45.543724    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"]
Sep 09 20:36:45.558782 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:45.558747    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:46.143784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:46.143724    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:50.280845 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:50.280787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:58.926823 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:58.926788    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:59.021667 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:36:59.021628    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:00.355826 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.355786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:00.563378 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.563334    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l" podUID="733c3c2b-b028-49a0-99f1-0fa2a475a16a" containerName="main" containerID="cri-o://287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20" gracePeriod=45
Sep 09 20:37:00.815551 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.815517    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:37:00.919698 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.919657    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-home\") pod \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") "
Sep 09 20:37:00.919909 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.919735    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-model-cache\") pod \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") "
Sep 09 20:37:00.919909 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.919793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tmp-dir\") pod \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") "
Sep 09 20:37:00.919909 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.919841    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s295\" (UniqueName: \"kubernetes.io/projected/733c3c2b-b028-49a0-99f1-0fa2a475a16a-kube-api-access-8s295\") pod \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") "
Sep 09 20:37:00.919909 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.919871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tls-certs\") pod \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") "
Sep 09 20:37:00.919909 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.919895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-dshm\") pod \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\" (UID: \"733c3c2b-b028-49a0-99f1-0fa2a475a16a\") "
Sep 09 20:37:00.921258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.920790    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-home" (OuterVolumeSpecName: "home") pod "733c3c2b-b028-49a0-99f1-0fa2a475a16a" (UID: "733c3c2b-b028-49a0-99f1-0fa2a475a16a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:00.921258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.921073    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "733c3c2b-b028-49a0-99f1-0fa2a475a16a" (UID: "733c3c2b-b028-49a0-99f1-0fa2a475a16a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:00.921258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.921217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-model-cache" (OuterVolumeSpecName: "model-cache") pod "733c3c2b-b028-49a0-99f1-0fa2a475a16a" (UID: "733c3c2b-b028-49a0-99f1-0fa2a475a16a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:00.923545 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.923512    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733c3c2b-b028-49a0-99f1-0fa2a475a16a-kube-api-access-8s295" (OuterVolumeSpecName: "kube-api-access-8s295") pod "733c3c2b-b028-49a0-99f1-0fa2a475a16a" (UID: "733c3c2b-b028-49a0-99f1-0fa2a475a16a"). InnerVolumeSpecName "kube-api-access-8s295". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:00.923769 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.923734    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-dshm" (OuterVolumeSpecName: "dshm") pod "733c3c2b-b028-49a0-99f1-0fa2a475a16a" (UID: "733c3c2b-b028-49a0-99f1-0fa2a475a16a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:00.923769 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:00.923751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "733c3c2b-b028-49a0-99f1-0fa2a475a16a" (UID: "733c3c2b-b028-49a0-99f1-0fa2a475a16a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:01.021429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.021343    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.021429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.021376    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.021429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.021386    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s295\" (UniqueName: \"kubernetes.io/projected/733c3c2b-b028-49a0-99f1-0fa2a475a16a-kube-api-access-8s295\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.021429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.021398    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/733c3c2b-b028-49a0-99f1-0fa2a475a16a-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.021429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.021407    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.021429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.021415    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/733c3c2b-b028-49a0-99f1-0fa2a475a16a-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.263149 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.263111    2578 generic.go:356] "Generic (PLEG): container finished" podID="733c3c2b-b028-49a0-99f1-0fa2a475a16a" containerID="287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20" exitCode=0
Sep 09 20:37:01.263336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.263151    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"
Sep 09 20:37:01.263336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.263155    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l" event={"ID":"733c3c2b-b028-49a0-99f1-0fa2a475a16a","Type":"ContainerDied","Data":"287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20"}
Sep 09 20:37:01.263336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.263277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l" event={"ID":"733c3c2b-b028-49a0-99f1-0fa2a475a16a","Type":"ContainerDied","Data":"337d258e2d56a0d760e0e993b1c4fc9a9457a2f1d0ad40adad6636eb750caea0"}
Sep 09 20:37:01.263336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.263304    2578 scope.go:117] "RemoveContainer" containerID="287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20"
Sep 09 20:37:01.272718 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.272688    2578 scope.go:117] "RemoveContainer" containerID="287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20"
Sep 09 20:37:01.272983 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:37:01.272964    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20\": container with ID starting with 287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20 not found: ID does not exist" containerID="287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20"
Sep 09 20:37:01.273039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.272990    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20"} err="failed to get container status \"287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20\": rpc error: code = NotFound desc = could not find container \"287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20\": container with ID starting with 287639a7664655a84e39b8b2236c2c43d128286da8211ace337364f51c84ad20 not found: ID does not exist"
Sep 09 20:37:01.283566 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.283544    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"]
Sep 09 20:37:01.287844 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.287823    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l"]
Sep 09 20:37:01.743430 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.743388    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-sg76l" podUID="733c3c2b-b028-49a0-99f1-0fa2a475a16a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded"
Sep 09 20:37:01.898429 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:01.898390    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="733c3c2b-b028-49a0-99f1-0fa2a475a16a" path="/var/lib/kubelet/pods/733c3c2b-b028-49a0-99f1-0fa2a475a16a/volumes"
Sep 09 20:37:04.853604 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:04.853572    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:10.470683 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:10.470641    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:10.538204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:10.538167    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:15.220946 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.220900    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"]
Sep 09 20:37:15.221416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.221217    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="733c3c2b-b028-49a0-99f1-0fa2a475a16a" containerName="main"
Sep 09 20:37:15.221416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.221227    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="733c3c2b-b028-49a0-99f1-0fa2a475a16a" containerName="main"
Sep 09 20:37:15.221416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.221236    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a64a19-6fa3-4acf-9763-0d0558f95c74" containerName="main"
Sep 09 20:37:15.221416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.221242    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a64a19-6fa3-4acf-9763-0d0558f95c74" containerName="main"
Sep 09 20:37:15.221416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.221290    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a64a19-6fa3-4acf-9763-0d0558f95c74" containerName="main"
Sep 09 20:37:15.221416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.221301    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="733c3c2b-b028-49a0-99f1-0fa2a475a16a" containerName="main"
Sep 09 20:37:15.225512 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.225481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.228004 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.227976    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 09 20:37:15.228940 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.228921    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 09 20:37:15.229011 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.228921    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-d9xsv\""
Sep 09 20:37:15.229063 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.228926    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 09 20:37:15.233830 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.233803    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"]
Sep 09 20:37:15.252583 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.252551    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:15.344319 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.344275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.344493 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.344332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.344493 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.344394    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-dshm\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.344493 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.344445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vrm\" (UniqueName: \"kubernetes.io/projected/e1eac92f-2eda-43a7-a22b-b9c0978c5381-kube-api-access-h5vrm\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.344626 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.344494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-model-cache\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.344626 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.344563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-home\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.445621 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.445580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vrm\" (UniqueName: \"kubernetes.io/projected/e1eac92f-2eda-43a7-a22b-b9c0978c5381-kube-api-access-h5vrm\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.445621 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.445620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-model-cache\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.445886 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.445653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-home\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.445886 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.445686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.445886 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.445712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.445886 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.445730    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-dshm\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.446105 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.446053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-model-cache\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.446105 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.446084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-home\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.446195 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.446129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.448062 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.448034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-dshm\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.448299 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.448281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.453675 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.453652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vrm\" (UniqueName: \"kubernetes.io/projected/e1eac92f-2eda-43a7-a22b-b9c0978c5381-kube-api-access-h5vrm\") pod \"owner-v2-kserve-5cf8959f95-g5ngr\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.537301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.537207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:15.810417 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:15.810388    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"]
Sep 09 20:37:15.812973 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:37:15.812941    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1eac92f_2eda_43a7_a22b_b9c0978c5381.slice/crio-dafb6fbf77c25975fc1855e47f6bfc7cb4ef8d0866aa05cdee445e9ebc5ef913 WatchSource:0}: Error finding container dafb6fbf77c25975fc1855e47f6bfc7cb4ef8d0866aa05cdee445e9ebc5ef913: Status 404 returned error can't find the container with id dafb6fbf77c25975fc1855e47f6bfc7cb4ef8d0866aa05cdee445e9ebc5ef913
Sep 09 20:37:16.334325 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:16.334283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr" event={"ID":"e1eac92f-2eda-43a7-a22b-b9c0978c5381","Type":"ContainerStarted","Data":"6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e"}
Sep 09 20:37:16.334733 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:16.334349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr" event={"ID":"e1eac92f-2eda-43a7-a22b-b9c0978c5381","Type":"ContainerStarted","Data":"dafb6fbf77c25975fc1855e47f6bfc7cb4ef8d0866aa05cdee445e9ebc5ef913"}
Sep 09 20:37:16.353102 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:16.353050    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr" podStartSLOduration=1.353035413 podStartE2EDuration="1.353035413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:16.351431939 +0000 UTC m=+2544.940085607" watchObservedRunningTime="2026-09-09 20:37:16.353035413 +0000 UTC m=+2544.941689069"
Sep 09 20:37:16.538109 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:16.538075    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:17.987577 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:17.987546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:24.015960 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:24.015926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:24.210427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:24.210391    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:25.537572 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:25.537536    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:25.554729 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:25.554697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:26.390491 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:26.390462    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:37:27.475379 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:27.475342    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:33.132665 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:33.132626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:33.281233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:33.281200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:54.605082 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:54.605052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:37:59.271410 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:37:59.271375    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:20.310299 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.310224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:20.413032 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.412994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:20.515330 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.515268    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:20.529033 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.529001    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"]
Sep 09 20:38:20.533557 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.533531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.536530 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.536501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 09 20:38:20.539110 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.539083    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:38:20.539669 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.539647    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 09 20:38:20.539804 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.539707    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-kxfpt\""
Sep 09 20:38:20.546907 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.546879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"]
Sep 09 20:38:20.609659 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.609620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.609843 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.609664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f835815-93b9-47dd-b64a-97a06a581d57-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.609843 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.609698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.609843 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.609724    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.609843 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.609789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.609843 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.609826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f835815-93b9-47dd-b64a-97a06a581d57-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.610116 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.609903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnr8\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-kube-api-access-xsnr8\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.610116 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.609944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.610116 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.609965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.610491 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.610470    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:20.711168 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnr8\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-kube-api-access-xsnr8\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711168 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f835815-93b9-47dd-b64a-97a06a581d57-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711264    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711282    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f835815-93b9-47dd-b64a-97a06a581d57-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711837 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-certs\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711897 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-data\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.711956 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-credential-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.712001 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.711976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-socket\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.712241 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.712213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f835815-93b9-47dd-b64a-97a06a581d57-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.713746 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.713723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-envoy\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.713847 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.713828    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f835815-93b9-47dd-b64a-97a06a581d57-istio-podinfo\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.723474 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.723450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-istio-token\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.723637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.723620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnr8\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-kube-api-access-xsnr8\") pod \"router-gateway-1-openshift-default-55f8b478f9-59scs\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:20.846177 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:20.846130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:21.144507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:21.144367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"]
Sep 09 20:38:21.149840 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:38:21.149785    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f835815_93b9_47dd_b64a_97a06a581d57.slice/crio-c17e090f0b481764435d43fdc0c867aeb37ca21ba9560567a28241aa9f2eadd8 WatchSource:0}: Error finding container c17e090f0b481764435d43fdc0c867aeb37ca21ba9560567a28241aa9f2eadd8: Status 404 returned error can't find the container with id c17e090f0b481764435d43fdc0c867aeb37ca21ba9560567a28241aa9f2eadd8
Sep 09 20:38:21.155835 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:21.152561    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:38:21.155835 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:21.152664    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:38:21.155835 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:21.152710    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:38:21.625389 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:21.625351    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" event={"ID":"7f835815-93b9-47dd-b64a-97a06a581d57","Type":"ContainerStarted","Data":"98e74e90702ee0d50713c8a361632523c4da007e1ee2fb494fa3768b383d8f7c"}
Sep 09 20:38:21.625781 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:21.625418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" event={"ID":"7f835815-93b9-47dd-b64a-97a06a581d57","Type":"ContainerStarted","Data":"c17e090f0b481764435d43fdc0c867aeb37ca21ba9560567a28241aa9f2eadd8"}
Sep 09 20:38:21.647748 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:21.647682    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" podStartSLOduration=1.6476599859999999 podStartE2EDuration="1.647659986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:21.646441538 +0000 UTC m=+2610.235095211" watchObservedRunningTime="2026-09-09 20:38:21.647659986 +0000 UTC m=+2610.236313651"
Sep 09 20:38:21.846671 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:21.846629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:22.851750 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:22.851718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:23.639112 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:23.636946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:23.640122 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:23.640097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:38:23.730591 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:23.730555    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:25.290039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:25.290000    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"]
Sep 09 20:38:25.343550 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:25.343517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:39.908097 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.908060    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"]
Sep 09 20:38:39.911286 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.911264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:39.914969 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.914944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-l6wdn\""
Sep 09 20:38:39.915109 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.915007    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 09 20:38:39.915109 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.915047    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 09 20:38:39.915109 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.915052    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 09 20:38:39.921943 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.921923    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"]
Sep 09 20:38:39.935174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.935148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:39.977701 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.977661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:39.977878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.977711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:39.977878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.977787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:39.977878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.977821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:39.977878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.977863    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxl8\" (UniqueName: \"kubernetes.io/projected/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-kube-api-access-mmxl8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:39.978081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:39.978009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.079170 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.079124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.079387 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.079296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.079387 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.079368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.079477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.079420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.079477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.079453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.079564 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.079496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxl8\" (UniqueName: \"kubernetes.io/projected/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-kube-api-access-mmxl8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.079662 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.079639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.079718 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.079683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.079756 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.079736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.081854 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.081832    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.082121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.082103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.095093 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.095066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxl8\" (UniqueName: \"kubernetes.io/projected/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-kube-api-access-mmxl8\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.139907 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.139872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:40.222203 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.222105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:40.309847 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.309808    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr" podUID="e1eac92f-2eda-43a7-a22b-b9c0978c5381" containerName="main" containerID="cri-o://6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e" gracePeriod=45
Sep 09 20:38:40.379882 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.379829    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr" podUID="e1eac92f-2eda-43a7-a22b-b9c0978c5381" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 09 20:38:40.501296 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.501263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"]
Sep 09 20:38:40.502379 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:38:40.502344    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc37a5ef_8707_4f87_acb3_5ad369cd3f4a.slice/crio-b5f006ccfd55ab942632761b75c9f32ca19882202dbf64c5acd114f52c3b6dcf WatchSource:0}: Error finding container b5f006ccfd55ab942632761b75c9f32ca19882202dbf64c5acd114f52c3b6dcf: Status 404 returned error can't find the container with id b5f006ccfd55ab942632761b75c9f32ca19882202dbf64c5acd114f52c3b6dcf
Sep 09 20:38:40.546560 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.546533    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:38:40.583176 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-model-cache\") pod \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") "
Sep 09 20:38:40.583345 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583196    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-dshm\") pod \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") "
Sep 09 20:38:40.583345 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-home\") pod \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") "
Sep 09 20:38:40.583345 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583286    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tmp-dir\") pod \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") "
Sep 09 20:38:40.583539 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583372    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tls-certs\") pod \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") "
Sep 09 20:38:40.583539 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583400    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5vrm\" (UniqueName: \"kubernetes.io/projected/e1eac92f-2eda-43a7-a22b-b9c0978c5381-kube-api-access-h5vrm\") pod \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\" (UID: \"e1eac92f-2eda-43a7-a22b-b9c0978c5381\") "
Sep 09 20:38:40.583652 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583558    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-home" (OuterVolumeSpecName: "home") pod "e1eac92f-2eda-43a7-a22b-b9c0978c5381" (UID: "e1eac92f-2eda-43a7-a22b-b9c0978c5381"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:40.583652 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583636    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1eac92f-2eda-43a7-a22b-b9c0978c5381" (UID: "e1eac92f-2eda-43a7-a22b-b9c0978c5381"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:40.584027 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583790    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.584027 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583810    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.584027 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.583868    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1eac92f-2eda-43a7-a22b-b9c0978c5381" (UID: "e1eac92f-2eda-43a7-a22b-b9c0978c5381"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:40.586076 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.586047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-dshm" (OuterVolumeSpecName: "dshm") pod "e1eac92f-2eda-43a7-a22b-b9c0978c5381" (UID: "e1eac92f-2eda-43a7-a22b-b9c0978c5381"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:40.586199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.586159    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1eac92f-2eda-43a7-a22b-b9c0978c5381" (UID: "e1eac92f-2eda-43a7-a22b-b9c0978c5381"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:40.586263 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.586214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1eac92f-2eda-43a7-a22b-b9c0978c5381-kube-api-access-h5vrm" (OuterVolumeSpecName: "kube-api-access-h5vrm") pod "e1eac92f-2eda-43a7-a22b-b9c0978c5381" (UID: "e1eac92f-2eda-43a7-a22b-b9c0978c5381"). InnerVolumeSpecName "kube-api-access-h5vrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:40.684199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.684152    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.684199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.684182    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1eac92f-2eda-43a7-a22b-b9c0978c5381-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.684199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.684191    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vrm\" (UniqueName: \"kubernetes.io/projected/e1eac92f-2eda-43a7-a22b-b9c0978c5381-kube-api-access-h5vrm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.684199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.684199    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1eac92f-2eda-43a7-a22b-b9c0978c5381-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:38:40.713127 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.713092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97" event={"ID":"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a","Type":"ContainerStarted","Data":"2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111"}
Sep 09 20:38:40.713415 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.713151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97" event={"ID":"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a","Type":"ContainerStarted","Data":"b5f006ccfd55ab942632761b75c9f32ca19882202dbf64c5acd114f52c3b6dcf"}
Sep 09 20:38:40.714420 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.714391    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1eac92f-2eda-43a7-a22b-b9c0978c5381" containerID="6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e" exitCode=0
Sep 09 20:38:40.714541 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.714434    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"
Sep 09 20:38:40.714541 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.714469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr" event={"ID":"e1eac92f-2eda-43a7-a22b-b9c0978c5381","Type":"ContainerDied","Data":"6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e"}
Sep 09 20:38:40.714541 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.714514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr" event={"ID":"e1eac92f-2eda-43a7-a22b-b9c0978c5381","Type":"ContainerDied","Data":"dafb6fbf77c25975fc1855e47f6bfc7cb4ef8d0866aa05cdee445e9ebc5ef913"}
Sep 09 20:38:40.714541 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.714536    2578 scope.go:117] "RemoveContainer" containerID="6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e"
Sep 09 20:38:40.723371 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.723354    2578 scope.go:117] "RemoveContainer" containerID="6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e"
Sep 09 20:38:40.723634 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:38:40.723613    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e\": container with ID starting with 6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e not found: ID does not exist" containerID="6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e"
Sep 09 20:38:40.723701 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.723640    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e"} err="failed to get container status \"6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e\": rpc error: code = NotFound desc = could not find container \"6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e\": container with ID starting with 6f6096c006120f9f43776f89ef26befd10356b393f7d1c9b2f363a99d82aaa4e not found: ID does not exist"
Sep 09 20:38:40.732331 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.732242    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97" podStartSLOduration=1.732229971 podStartE2EDuration="1.732229971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:40.73078703 +0000 UTC m=+2629.319440686" watchObservedRunningTime="2026-09-09 20:38:40.732229971 +0000 UTC m=+2629.320883627"
Sep 09 20:38:40.753618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.753580    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"]
Sep 09 20:38:40.759729 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:40.759700    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-g5ngr"]
Sep 09 20:38:41.222424 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:41.222382    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:41.897706 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:41.897658    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1eac92f-2eda-43a7-a22b-b9c0978c5381" path="/var/lib/kubelet/pods/e1eac92f-2eda-43a7-a22b-b9c0978c5381/volumes"
Sep 09 20:38:43.026338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:43.026283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:50.222617 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:50.222572    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:50.235159 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:50.235126    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:50.771101 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:50.771071    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:38:51.753139 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:38:51.753105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:01.635752 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:01.635715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:02.414469 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:02.414435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:02.816592 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:02.816503    2578 generic.go:356] "Generic (PLEG): container finished" podID="7f835815-93b9-47dd-b64a-97a06a581d57" containerID="98e74e90702ee0d50713c8a361632523c4da007e1ee2fb494fa3768b383d8f7c" exitCode=0
Sep 09 20:39:02.816592 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:02.816577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" event={"ID":"7f835815-93b9-47dd-b64a-97a06a581d57","Type":"ContainerDied","Data":"98e74e90702ee0d50713c8a361632523c4da007e1ee2fb494fa3768b383d8f7c"}
Sep 09 20:39:02.817038 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:02.816971    2578 scope.go:117] "RemoveContainer" containerID="98e74e90702ee0d50713c8a361632523c4da007e1ee2fb494fa3768b383d8f7c"
Sep 09 20:39:02.817876 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:02.817844    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:39:02.817926 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:02.817916    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:39:02.817968 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:02.817953    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:39:03.823431 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:03.823392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" event={"ID":"7f835815-93b9-47dd-b64a-97a06a581d57","Type":"ContainerStarted","Data":"86163b0e784ff97d5402d09d78f927034e043abd913beb42f96c8a2a9675c1e4"}
Sep 09 20:39:03.849865 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:03.849831    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:39:03.850170 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:03.850119    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 20:39:04.847789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:04.847728    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 20:39:05.846953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:05.846898    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 20:39:06.847592 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:06.847547    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 20:39:07.847521 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:07.847476    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 20:39:08.001412 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:08.001372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:08.062953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:08.062910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:08.558300 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:08.558260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:08.636725 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:08.636686    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:39:08.846637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:08.846569    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 09 20:39:09.850485 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:09.850449    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:39:09.851570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:09.851549    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:39:14.176674 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.176642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:14.184449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.184419    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"]
Sep 09 20:39:14.195130 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:39:14.195093    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:39:14.195130 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:39:14.195130 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:39:14.195130 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111" cmd=["/bin/sleep","15"]
Sep 09 20:39:14.195473 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.195154    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97" message="" err=<
Sep 09 20:39:14.195473 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:39:14.195473 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:39:14.195473 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:39:14.195473 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:39:14.195174    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:39:14.195473 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:39:14.195473 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:39:14.195473 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97" podUID="cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" containerName="main" containerID="cri-o://2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111"
Sep 09 20:39:14.195473 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.195217    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97" podUID="cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" containerName="main" containerID="cri-o://2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111" gracePeriod=60
Sep 09 20:39:14.240478 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.240447    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:39:14.541581 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.541552    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:39:14.678906 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.678874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmxl8\" (UniqueName: \"kubernetes.io/projected/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-kube-api-access-mmxl8\") pod \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") "
Sep 09 20:39:14.679086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.678926    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-dshm\") pod \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") "
Sep 09 20:39:14.679086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.678944    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-model-cache\") pod \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") "
Sep 09 20:39:14.679086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.678983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-home\") pod \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") "
Sep 09 20:39:14.679086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.679051    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tmp-dir\") pod \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") "
Sep 09 20:39:14.679086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.679069    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tls-certs\") pod \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\" (UID: \"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a\") "
Sep 09 20:39:14.679397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.679266    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" (UID: "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:14.679397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.679290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-home" (OuterVolumeSpecName: "home") pod "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" (UID: "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:14.679397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.679319    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" (UID: "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:14.681271 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.681241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" (UID: "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:39:14.681395 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.681301    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-dshm" (OuterVolumeSpecName: "dshm") pod "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" (UID: "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:14.681395 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.681322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-kube-api-access-mmxl8" (OuterVolumeSpecName: "kube-api-access-mmxl8") pod "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" (UID: "cc37a5ef-8707-4f87-acb3-5ad369cd3f4a"). InnerVolumeSpecName "kube-api-access-mmxl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:39:14.779809 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.779725    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxl8\" (UniqueName: \"kubernetes.io/projected/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-kube-api-access-mmxl8\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:39:14.779809 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.779764    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:39:14.779809 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.779781    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:39:14.779809 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.779799    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:39:14.780049 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.779813    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:39:14.780049 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.779827    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:39:14.872345 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.872292    2578 generic.go:356] "Generic (PLEG): container finished" podID="cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" containerID="2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111" exitCode=0
Sep 09 20:39:14.872540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.872358    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"
Sep 09 20:39:14.872540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.872375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97" event={"ID":"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a","Type":"ContainerDied","Data":"2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111"}
Sep 09 20:39:14.872540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.872430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97" event={"ID":"cc37a5ef-8707-4f87-acb3-5ad369cd3f4a","Type":"ContainerDied","Data":"b5f006ccfd55ab942632761b75c9f32ca19882202dbf64c5acd114f52c3b6dcf"}
Sep 09 20:39:14.872540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.872453    2578 scope.go:117] "RemoveContainer" containerID="2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111"
Sep 09 20:39:14.882360 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.882337    2578 scope.go:117] "RemoveContainer" containerID="2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111"
Sep 09 20:39:14.882680 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:39:14.882657    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111\": container with ID starting with 2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111 not found: ID does not exist" containerID="2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111"
Sep 09 20:39:14.882762 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.882689    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111"} err="failed to get container status \"2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111\": rpc error: code = NotFound desc = could not find container \"2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111\": container with ID starting with 2535426ebd297b64843881cfa1a37a03dbdbf08badb6bbbb24b9ac1acf961111 not found: ID does not exist"
Sep 09 20:39:14.896419 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.896383    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"]
Sep 09 20:39:14.901232 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:14.901196    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995bwg97"]
Sep 09 20:39:15.895477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:15.895444    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" path="/var/lib/kubelet/pods/cc37a5ef-8707-4f87-acb3-5ad369cd3f4a/volumes"
Sep 09 20:39:52.050430 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:52.050400    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:39:52.058809 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:39:52.058782    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:40:04.340505 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:04.340472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:04.402147 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:04.402119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:08.154945 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:08.154898    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:12.990085 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:12.990048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:13.090379 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:13.090344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:17.941716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:17.941671    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"]
Sep 09 20:40:17.942221 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:17.941943    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy" containerID="cri-o://86163b0e784ff97d5402d09d78f927034e043abd913beb42f96c8a2a9675c1e4" gracePeriod=30
Sep 09 20:40:17.994549 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:17.994508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:23.174732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.174692    2578 generic.go:356] "Generic (PLEG): container finished" podID="7f835815-93b9-47dd-b64a-97a06a581d57" containerID="86163b0e784ff97d5402d09d78f927034e043abd913beb42f96c8a2a9675c1e4" exitCode=0
Sep 09 20:40:23.175143 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.174774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" event={"ID":"7f835815-93b9-47dd-b64a-97a06a581d57","Type":"ContainerDied","Data":"86163b0e784ff97d5402d09d78f927034e043abd913beb42f96c8a2a9675c1e4"}
Sep 09 20:40:23.175143 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.174851    2578 scope.go:117] "RemoveContainer" containerID="98e74e90702ee0d50713c8a361632523c4da007e1ee2fb494fa3768b383d8f7c"
Sep 09 20:40:23.206449 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.206421    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:40:23.369947 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.369913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-envoy\") pod \"7f835815-93b9-47dd-b64a-97a06a581d57\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") "
Sep 09 20:40:23.369947 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.369949    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f835815-93b9-47dd-b64a-97a06a581d57-istiod-ca-cert\") pod \"7f835815-93b9-47dd-b64a-97a06a581d57\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") "
Sep 09 20:40:23.370199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.369972    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsnr8\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-kube-api-access-xsnr8\") pod \"7f835815-93b9-47dd-b64a-97a06a581d57\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") "
Sep 09 20:40:23.370199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-istio-token\") pod \"7f835815-93b9-47dd-b64a-97a06a581d57\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") "
Sep 09 20:40:23.370199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370040    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f835815-93b9-47dd-b64a-97a06a581d57-istio-podinfo\") pod \"7f835815-93b9-47dd-b64a-97a06a581d57\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") "
Sep 09 20:40:23.370199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370059    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-credential-socket\") pod \"7f835815-93b9-47dd-b64a-97a06a581d57\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") "
Sep 09 20:40:23.370199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370102    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-data\") pod \"7f835815-93b9-47dd-b64a-97a06a581d57\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") "
Sep 09 20:40:23.370199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370142    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-certs\") pod \"7f835815-93b9-47dd-b64a-97a06a581d57\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") "
Sep 09 20:40:23.370199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370168    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-socket\") pod \"7f835815-93b9-47dd-b64a-97a06a581d57\" (UID: \"7f835815-93b9-47dd-b64a-97a06a581d57\") "
Sep 09 20:40:23.370596 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f835815-93b9-47dd-b64a-97a06a581d57-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f835815-93b9-47dd-b64a-97a06a581d57" (UID: "7f835815-93b9-47dd-b64a-97a06a581d57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:23.370596 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370545    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f835815-93b9-47dd-b64a-97a06a581d57-istiod-ca-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:40:23.370957 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f835815-93b9-47dd-b64a-97a06a581d57" (UID: "7f835815-93b9-47dd-b64a-97a06a581d57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:23.371067 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370947    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f835815-93b9-47dd-b64a-97a06a581d57" (UID: "7f835815-93b9-47dd-b64a-97a06a581d57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:23.371067 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370975    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f835815-93b9-47dd-b64a-97a06a581d57" (UID: "7f835815-93b9-47dd-b64a-97a06a581d57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:23.371067 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.370934    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f835815-93b9-47dd-b64a-97a06a581d57" (UID: "7f835815-93b9-47dd-b64a-97a06a581d57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:23.372558 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.372519    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f835815-93b9-47dd-b64a-97a06a581d57-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f835815-93b9-47dd-b64a-97a06a581d57" (UID: "7f835815-93b9-47dd-b64a-97a06a581d57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:40:23.372657 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.372612    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f835815-93b9-47dd-b64a-97a06a581d57" (UID: "7f835815-93b9-47dd-b64a-97a06a581d57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:23.372729 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.372707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-kube-api-access-xsnr8" (OuterVolumeSpecName: "kube-api-access-xsnr8") pod "7f835815-93b9-47dd-b64a-97a06a581d57" (UID: "7f835815-93b9-47dd-b64a-97a06a581d57"). InnerVolumeSpecName "kube-api-access-xsnr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:23.372842 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.372822    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f835815-93b9-47dd-b64a-97a06a581d57" (UID: "7f835815-93b9-47dd-b64a-97a06a581d57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:23.471403 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.471362    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsnr8\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-kube-api-access-xsnr8\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:40:23.471403 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.471396    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f835815-93b9-47dd-b64a-97a06a581d57-istio-token\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:40:23.471403 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.471409    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f835815-93b9-47dd-b64a-97a06a581d57-istio-podinfo\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:40:23.471649 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.471422    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-credential-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:40:23.471649 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.471433    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-data\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:40:23.471649 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.471444    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:40:23.471649 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.471455    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-workload-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:40:23.471649 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:23.471468    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f835815-93b9-47dd-b64a-97a06a581d57-istio-envoy\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:40:24.181091 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.180996    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"
Sep 09 20:40:24.181091 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.181015    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs" event={"ID":"7f835815-93b9-47dd-b64a-97a06a581d57","Type":"ContainerDied","Data":"c17e090f0b481764435d43fdc0c867aeb37ca21ba9560567a28241aa9f2eadd8"}
Sep 09 20:40:24.181091 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.181069    2578 scope.go:117] "RemoveContainer" containerID="86163b0e784ff97d5402d09d78f927034e043abd913beb42f96c8a2a9675c1e4"
Sep 09 20:40:24.199529 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.199503    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"]
Sep 09 20:40:24.202696 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.202669    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-55f8b478f9-59scs"]
Sep 09 20:40:24.752321 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.752277    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:24.981438 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981404    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"]
Sep 09 20:40:24.981750 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981735    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" containerName="main"
Sep 09 20:40:24.981818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981752    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" containerName="main"
Sep 09 20:40:24.981818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981761    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1eac92f-2eda-43a7-a22b-b9c0978c5381" containerName="main"
Sep 09 20:40:24.981818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981767    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1eac92f-2eda-43a7-a22b-b9c0978c5381" containerName="main"
Sep 09 20:40:24.981818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981775    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy"
Sep 09 20:40:24.981818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981781    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy"
Sep 09 20:40:24.981818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981790    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy"
Sep 09 20:40:24.981818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981796    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy"
Sep 09 20:40:24.982134 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981856    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc37a5ef-8707-4f87-acb3-5ad369cd3f4a" containerName="main"
Sep 09 20:40:24.982134 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981863    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1eac92f-2eda-43a7-a22b-b9c0978c5381" containerName="main"
Sep 09 20:40:24.982134 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981872    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy"
Sep 09 20:40:24.982134 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.981881    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" containerName="istio-proxy"
Sep 09 20:40:24.986178 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.986154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:40:24.988859 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.988808    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 09 20:40:24.989014 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.988936    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 09 20:40:24.989014 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.988966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-znvvn\""
Sep 09 20:40:24.989907 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.989856    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 09 20:40:24.990027 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.989919    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-f468r\""
Sep 09 20:40:24.995190 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:24.995168    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"]
Sep 09 20:40:25.022740 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.022662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:25.086900 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.086865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a94e46-de89-4e82-ab53-b7f5fcf61661-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25\" (UID: \"47a94e46-de89-4e82-ab53-b7f5fcf61661\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:40:25.087046 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.086906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqhmm\" (UniqueName: \"kubernetes.io/projected/47a94e46-de89-4e82-ab53-b7f5fcf61661-kube-api-access-fqhmm\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25\" (UID: \"47a94e46-de89-4e82-ab53-b7f5fcf61661\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:40:25.188030 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.187995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a94e46-de89-4e82-ab53-b7f5fcf61661-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25\" (UID: \"47a94e46-de89-4e82-ab53-b7f5fcf61661\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:40:25.188030 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.188036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqhmm\" (UniqueName: \"kubernetes.io/projected/47a94e46-de89-4e82-ab53-b7f5fcf61661-kube-api-access-fqhmm\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25\" (UID: \"47a94e46-de89-4e82-ab53-b7f5fcf61661\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:40:25.190539 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.190517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a94e46-de89-4e82-ab53-b7f5fcf61661-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25\" (UID: \"47a94e46-de89-4e82-ab53-b7f5fcf61661\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:40:25.198133 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.198102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqhmm\" (UniqueName: \"kubernetes.io/projected/47a94e46-de89-4e82-ab53-b7f5fcf61661-kube-api-access-fqhmm\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25\" (UID: \"47a94e46-de89-4e82-ab53-b7f5fcf61661\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:40:25.298098 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.298007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:40:25.569561 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.569531    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"]
Sep 09 20:40:25.571513 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:40:25.571487    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a94e46_de89_4e82_ab53_b7f5fcf61661.slice/crio-2b6b68935ca6f3f26b140b6c8f4ce3a0061bcbe36027b8e4c17619ab62a52c2b WatchSource:0}: Error finding container 2b6b68935ca6f3f26b140b6c8f4ce3a0061bcbe36027b8e4c17619ab62a52c2b: Status 404 returned error can't find the container with id 2b6b68935ca6f3f26b140b6c8f4ce3a0061bcbe36027b8e4c17619ab62a52c2b
Sep 09 20:40:25.897195 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:25.897159    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f835815-93b9-47dd-b64a-97a06a581d57" path="/var/lib/kubelet/pods/7f835815-93b9-47dd-b64a-97a06a581d57/volumes"
Sep 09 20:40:26.195910 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:26.195811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" event={"ID":"47a94e46-de89-4e82-ab53-b7f5fcf61661","Type":"ContainerStarted","Data":"0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564"}
Sep 09 20:40:26.195910 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:26.195894    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:40:26.196395 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:26.195921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" event={"ID":"47a94e46-de89-4e82-ab53-b7f5fcf61661","Type":"ContainerStarted","Data":"2b6b68935ca6f3f26b140b6c8f4ce3a0061bcbe36027b8e4c17619ab62a52c2b"}
Sep 09 20:40:26.211956 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:26.211909    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" podStartSLOduration=2.211891896 podStartE2EDuration="2.211891896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:26.210333487 +0000 UTC m=+2734.798987135" watchObservedRunningTime="2026-09-09 20:40:26.211891896 +0000 UTC m=+2734.800545553"
Sep 09 20:40:40.369665 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:40.369626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:40.498155 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:40.498124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:43.533586 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:43.533535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:46.011042 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:46.011002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:54.434556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:54.434519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:54.536604 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:54.536567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:57.203109 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:40:57.203075    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:41:01.145002 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:01.144962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:03.167349 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:03.167295    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:03.434145 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:03.434062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:06.774967 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:06.774930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:06.845688 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:06.845656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:09.758601 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:09.758569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:15.396633 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:15.396545    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:16.565799 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:16.565760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:25.056478 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:25.056439    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:25.126796 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:25.126759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:33.728628 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:33.728593    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:33.829564 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:33.829531    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:43.330338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.330284    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"]
Sep 09 20:41:43.333388 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.333364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.335786 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.335759    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 09 20:41:43.336741 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.336713    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 09 20:41:43.336848 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.336715    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 09 20:41:43.336848 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.336822    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-9wq6g\""
Sep 09 20:41:43.342461 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.342434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"]
Sep 09 20:41:43.414089 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.414048    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:43.425691 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.425663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.425851 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.425702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.425851 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.425767    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.425851 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.425821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b1a3-5830-4eae-a04e-6cf61624935a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.426034 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.425857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8ng\" (UniqueName: \"kubernetes.io/projected/45d4b1a3-5830-4eae-a04e-6cf61624935a-kube-api-access-mt8ng\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.426034 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.425899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.526955 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.526921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.527140 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.526974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b1a3-5830-4eae-a04e-6cf61624935a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.527140 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.527012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8ng\" (UniqueName: \"kubernetes.io/projected/45d4b1a3-5830-4eae-a04e-6cf61624935a-kube-api-access-mt8ng\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.527140 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.527038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.527140 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.527074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.527140 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.527114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.527437 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.527415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.527500 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.527486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.527556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.527535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.529655 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.529632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b1a3-5830-4eae-a04e-6cf61624935a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.529771 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.529723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.535547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.535525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8ng\" (UniqueName: \"kubernetes.io/projected/45d4b1a3-5830-4eae-a04e-6cf61624935a-kube-api-access-mt8ng\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.643361 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.643300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:43.644732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.644711    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:43.916127 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:43.916045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"]
Sep 09 20:41:43.919148 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:41:43.919118    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d4b1a3_5830_4eae_a04e_6cf61624935a.slice/crio-de9ac5d1fa41c9f8bf6c9f33ded1575c9932c1d268c8b3d6d8abf783c5c8b2cd WatchSource:0}: Error finding container de9ac5d1fa41c9f8bf6c9f33ded1575c9932c1d268c8b3d6d8abf783c5c8b2cd: Status 404 returned error can't find the container with id de9ac5d1fa41c9f8bf6c9f33ded1575c9932c1d268c8b3d6d8abf783c5c8b2cd
Sep 09 20:41:44.540498 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:44.540458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8" event={"ID":"45d4b1a3-5830-4eae-a04e-6cf61624935a","Type":"ContainerStarted","Data":"162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380"}
Sep 09 20:41:44.540965 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:44.540527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8" event={"ID":"45d4b1a3-5830-4eae-a04e-6cf61624935a","Type":"ContainerStarted","Data":"de9ac5d1fa41c9f8bf6c9f33ded1575c9932c1d268c8b3d6d8abf783c5c8b2cd"}
Sep 09 20:41:44.582426 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:44.582357    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8" podStartSLOduration=1.5823346090000001 podStartE2EDuration="1.582334609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:44.580150527 +0000 UTC m=+2813.168804185" watchObservedRunningTime="2026-09-09 20:41:44.582334609 +0000 UTC m=+2813.170988265"
Sep 09 20:41:44.644529 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:44.644488    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:51.100364 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:51.100326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:53.644221 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:53.644171    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:53.656794 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:53.656766    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:54.594743 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:54.594704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:41:59.932459 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:41:59.932417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:17.266297 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.266258    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"]
Sep 09 20:42:17.268569 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.268524    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:17.277109 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:42:17.277061    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:42:17.277109 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:42:17.277109 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:42:17.277109 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380" cmd=["/bin/sleep","15"]
Sep 09 20:42:17.277450 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.277120    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8" message="" err=<
Sep 09 20:42:17.277450 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:42:17.277450 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:42:17.277450 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 20:42:17.277450 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:42:17.277133    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:42:17.277450 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:42:17.277450 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 20:42:17.277450 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8" podUID="45d4b1a3-5830-4eae-a04e-6cf61624935a" containerName="main" containerID="cri-o://162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380"
Sep 09 20:42:17.277450 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.277176    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8" podUID="45d4b1a3-5830-4eae-a04e-6cf61624935a" containerName="main" containerID="cri-o://162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380" gracePeriod=60
Sep 09 20:42:17.332045 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.332015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:17.633297 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.633267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:42:17.682939 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.682901    2578 generic.go:356] "Generic (PLEG): container finished" podID="45d4b1a3-5830-4eae-a04e-6cf61624935a" containerID="162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380" exitCode=0
Sep 09 20:42:17.683193 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.682948    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"
Sep 09 20:42:17.683193 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.683033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8" event={"ID":"45d4b1a3-5830-4eae-a04e-6cf61624935a","Type":"ContainerDied","Data":"162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380"}
Sep 09 20:42:17.683193 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.683096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8" event={"ID":"45d4b1a3-5830-4eae-a04e-6cf61624935a","Type":"ContainerDied","Data":"de9ac5d1fa41c9f8bf6c9f33ded1575c9932c1d268c8b3d6d8abf783c5c8b2cd"}
Sep 09 20:42:17.683193 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.683129    2578 scope.go:117] "RemoveContainer" containerID="162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380"
Sep 09 20:42:17.693251 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.693227    2578 scope.go:117] "RemoveContainer" containerID="162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380"
Sep 09 20:42:17.693591 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:42:17.693566    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380\": container with ID starting with 162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380 not found: ID does not exist" containerID="162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380"
Sep 09 20:42:17.693654 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.693608    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380"} err="failed to get container status \"162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380\": rpc error: code = NotFound desc = could not find container \"162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380\": container with ID starting with 162ffa52d831178b83f6a3c568668ff41d544b94195df2de93c0e67b5f289380 not found: ID does not exist"
Sep 09 20:42:17.704011 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.703984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-tmp-dir\") pod \"45d4b1a3-5830-4eae-a04e-6cf61624935a\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") "
Sep 09 20:42:17.704109 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.704029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt8ng\" (UniqueName: \"kubernetes.io/projected/45d4b1a3-5830-4eae-a04e-6cf61624935a-kube-api-access-mt8ng\") pod \"45d4b1a3-5830-4eae-a04e-6cf61624935a\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") "
Sep 09 20:42:17.704109 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.704063    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-dshm\") pod \"45d4b1a3-5830-4eae-a04e-6cf61624935a\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") "
Sep 09 20:42:17.704109 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.704081    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-home\") pod \"45d4b1a3-5830-4eae-a04e-6cf61624935a\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") "
Sep 09 20:42:17.704271 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.704110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b1a3-5830-4eae-a04e-6cf61624935a-tls-certs\") pod \"45d4b1a3-5830-4eae-a04e-6cf61624935a\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") "
Sep 09 20:42:17.704271 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.704139    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-model-cache\") pod \"45d4b1a3-5830-4eae-a04e-6cf61624935a\" (UID: \"45d4b1a3-5830-4eae-a04e-6cf61624935a\") "
Sep 09 20:42:17.704416 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.704338    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45d4b1a3-5830-4eae-a04e-6cf61624935a" (UID: "45d4b1a3-5830-4eae-a04e-6cf61624935a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:17.704474 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.704420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-home" (OuterVolumeSpecName: "home") pod "45d4b1a3-5830-4eae-a04e-6cf61624935a" (UID: "45d4b1a3-5830-4eae-a04e-6cf61624935a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:17.704538 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.704494    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:42:17.704586 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.704562    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-model-cache" (OuterVolumeSpecName: "model-cache") pod "45d4b1a3-5830-4eae-a04e-6cf61624935a" (UID: "45d4b1a3-5830-4eae-a04e-6cf61624935a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:17.706280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.706258    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-dshm" (OuterVolumeSpecName: "dshm") pod "45d4b1a3-5830-4eae-a04e-6cf61624935a" (UID: "45d4b1a3-5830-4eae-a04e-6cf61624935a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:17.706466 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.706440    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d4b1a3-5830-4eae-a04e-6cf61624935a-kube-api-access-mt8ng" (OuterVolumeSpecName: "kube-api-access-mt8ng") pod "45d4b1a3-5830-4eae-a04e-6cf61624935a" (UID: "45d4b1a3-5830-4eae-a04e-6cf61624935a"). InnerVolumeSpecName "kube-api-access-mt8ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:42:17.706630 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.706603    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45d4b1a3-5830-4eae-a04e-6cf61624935a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45d4b1a3-5830-4eae-a04e-6cf61624935a" (UID: "45d4b1a3-5830-4eae-a04e-6cf61624935a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:42:17.805356 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.805249    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt8ng\" (UniqueName: \"kubernetes.io/projected/45d4b1a3-5830-4eae-a04e-6cf61624935a-kube-api-access-mt8ng\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:42:17.805356 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.805279    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:42:17.805356 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.805288    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:42:17.805356 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.805297    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45d4b1a3-5830-4eae-a04e-6cf61624935a-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:42:17.805356 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.805323    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45d4b1a3-5830-4eae-a04e-6cf61624935a-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:42:17.998372 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:17.998342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"]
Sep 09 20:42:18.004609 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:18.004583    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8"]
Sep 09 20:42:18.583826 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:18.583784    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-67cm8" podUID="45d4b1a3-5830-4eae-a04e-6cf61624935a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:42:19.896290 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:19.896256    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d4b1a3-5830-4eae-a04e-6cf61624935a" path="/var/lib/kubelet/pods/45d4b1a3-5830-4eae-a04e-6cf61624935a/volumes"
Sep 09 20:42:40.390362 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:40.390322    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"]
Sep 09 20:42:40.412262 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:40.412225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:40.469151 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:40.469120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:47.202639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:47.202544    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" podUID="47a94e46-de89-4e82-ab53-b7f5fcf61661" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:42:55.410256 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:55.410205    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" podUID="47a94e46-de89-4e82-ab53-b7f5fcf61661" containerName="main" containerID="cri-o://0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564" gracePeriod=45
Sep 09 20:42:57.202225 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:42:57.202181    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" podUID="47a94e46-de89-4e82-ab53-b7f5fcf61661" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:43:07.202825 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:07.202780    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" podUID="47a94e46-de89-4e82-ab53-b7f5fcf61661" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:43:07.203215 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:07.202895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:43:07.383240 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:07.383208    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:07.456059 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:07.455965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:17.202112 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:17.202066    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" podUID="47a94e46-de89-4e82-ab53-b7f5fcf61661" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:43:25.663903 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.663872    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:43:25.731976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.731881    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a94e46-de89-4e82-ab53-b7f5fcf61661-tls-certs\") pod \"47a94e46-de89-4e82-ab53-b7f5fcf61661\" (UID: \"47a94e46-de89-4e82-ab53-b7f5fcf61661\") "
Sep 09 20:43:25.731976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.731946    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqhmm\" (UniqueName: \"kubernetes.io/projected/47a94e46-de89-4e82-ab53-b7f5fcf61661-kube-api-access-fqhmm\") pod \"47a94e46-de89-4e82-ab53-b7f5fcf61661\" (UID: \"47a94e46-de89-4e82-ab53-b7f5fcf61661\") "
Sep 09 20:43:25.734220 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.734183    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a94e46-de89-4e82-ab53-b7f5fcf61661-kube-api-access-fqhmm" (OuterVolumeSpecName: "kube-api-access-fqhmm") pod "47a94e46-de89-4e82-ab53-b7f5fcf61661" (UID: "47a94e46-de89-4e82-ab53-b7f5fcf61661"). InnerVolumeSpecName "kube-api-access-fqhmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:25.734220 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.734211    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47a94e46-de89-4e82-ab53-b7f5fcf61661-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47a94e46-de89-4e82-ab53-b7f5fcf61661" (UID: "47a94e46-de89-4e82-ab53-b7f5fcf61661"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:25.832993 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.832956    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47a94e46-de89-4e82-ab53-b7f5fcf61661-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:43:25.832993 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.832990    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqhmm\" (UniqueName: \"kubernetes.io/projected/47a94e46-de89-4e82-ab53-b7f5fcf61661-kube-api-access-fqhmm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:43:25.981962 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.981921    2578 generic.go:356] "Generic (PLEG): container finished" podID="47a94e46-de89-4e82-ab53-b7f5fcf61661" containerID="0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564" exitCode=0
Sep 09 20:43:25.982151 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.981975    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"
Sep 09 20:43:25.982151 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.982004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" event={"ID":"47a94e46-de89-4e82-ab53-b7f5fcf61661","Type":"ContainerDied","Data":"0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564"}
Sep 09 20:43:25.982151 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.982059    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25" event={"ID":"47a94e46-de89-4e82-ab53-b7f5fcf61661","Type":"ContainerDied","Data":"2b6b68935ca6f3f26b140b6c8f4ce3a0061bcbe36027b8e4c17619ab62a52c2b"}
Sep 09 20:43:25.982151 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.982081    2578 scope.go:117] "RemoveContainer" containerID="0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564"
Sep 09 20:43:25.990363 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.990345    2578 scope.go:117] "RemoveContainer" containerID="0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564"
Sep 09 20:43:25.990648 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:43:25.990629    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564\": container with ID starting with 0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564 not found: ID does not exist" containerID="0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564"
Sep 09 20:43:25.990692 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.990656    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564"} err="failed to get container status \"0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564\": rpc error: code = NotFound desc = could not find container \"0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564\": container with ID starting with 0f7d64f9c2a3f86efce48aa8086895b1d38e86a24da0c15a5f8d5fcbfa55c564 not found: ID does not exist"
Sep 09 20:43:25.997498 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:25.997477    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"]
Sep 09 20:43:26.001127 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:26.001107    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewvx25"]
Sep 09 20:43:27.896142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:27.896107    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47a94e46-de89-4e82-ab53-b7f5fcf61661" path="/var/lib/kubelet/pods/47a94e46-de89-4e82-ab53-b7f5fcf61661/volumes"
Sep 09 20:43:30.528417 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:30.528381    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:30.629012 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:30.628981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:36.531352 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:36.531282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:42.361950 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:42.361913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:42.442221 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:42.442188    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:57.476998 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:57.476961    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:57.572637 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:43:57.572600    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:00.815712 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.815677    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"]
Sep 09 20:44:00.816157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.816002    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47a94e46-de89-4e82-ab53-b7f5fcf61661" containerName="main"
Sep 09 20:44:00.816157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.816014    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47a94e46-de89-4e82-ab53-b7f5fcf61661" containerName="main"
Sep 09 20:44:00.816157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.816028    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d4b1a3-5830-4eae-a04e-6cf61624935a" containerName="main"
Sep 09 20:44:00.816157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.816034    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d4b1a3-5830-4eae-a04e-6cf61624935a" containerName="main"
Sep 09 20:44:00.816157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.816087    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d4b1a3-5830-4eae-a04e-6cf61624935a" containerName="main"
Sep 09 20:44:00.816157 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.816095    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47a94e46-de89-4e82-ab53-b7f5fcf61661" containerName="main"
Sep 09 20:44:00.819270 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.819247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:00.822368 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.822338    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-vdznc\""
Sep 09 20:44:00.822541 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.822388    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 09 20:44:00.822541 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.822445    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 09 20:44:00.822663 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.822638    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 09 20:44:00.830566 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.830541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"]
Sep 09 20:44:00.858187 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.858144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:00.937229 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.937185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:00.937229 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.937230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:00.937456 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.937250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:00.937456 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.937380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxkk\" (UniqueName: \"kubernetes.io/projected/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kube-api-access-mbxkk\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:00.937456 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.937410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:00.937456 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.937433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:00.937593 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:00.937482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.038520 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.038484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxkk\" (UniqueName: \"kubernetes.io/projected/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kube-api-access-mbxkk\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.038728 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.038527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.038728 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.038548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.038728 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.038577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.038728 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.038613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.038728 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.038639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.038728 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.038670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.039101 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.039068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.039240 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.039196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.039299 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.039256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.039496 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.039472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.041301 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.041279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.041412 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.041360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.046670 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.046637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxkk\" (UniqueName: \"kubernetes.io/projected/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kube-api-access-mbxkk\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.109675 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.109637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:01.131596 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.131564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:01.409499 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.409470    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"]
Sep 09 20:44:01.459380 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:01.459351    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:44:02.141199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:02.141165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" event={"ID":"a1fc06bf-5cc6-4c31-9949-19e4333c2f94","Type":"ContainerStarted","Data":"f9186c23551eb88b83a7b8e0c543ca8f6a791867622c44494415cbf21ca555cf"}
Sep 09 20:44:04.184797 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:04.184749    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:09.749265 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:09.749148    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:09.850396 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:09.850363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:10.205588 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:10.205550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" event={"ID":"a1fc06bf-5cc6-4c31-9949-19e4333c2f94","Type":"ContainerStarted","Data":"6999117f5b838002cbab8f9f0bf27e9425d18da16ce5d3c29b450a500305c70f"}
Sep 09 20:44:11.211697 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:11.211650    2578 generic.go:356] "Generic (PLEG): container finished" podID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerID="6999117f5b838002cbab8f9f0bf27e9425d18da16ce5d3c29b450a500305c70f" exitCode=0
Sep 09 20:44:11.212091 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:11.211732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" event={"ID":"a1fc06bf-5cc6-4c31-9949-19e4333c2f94","Type":"ContainerDied","Data":"6999117f5b838002cbab8f9f0bf27e9425d18da16ce5d3c29b450a500305c70f"}
Sep 09 20:44:12.219141 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:12.219052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" event={"ID":"a1fc06bf-5cc6-4c31-9949-19e4333c2f94","Type":"ContainerStarted","Data":"1b0528c4b6bdf204d338328b07602b16275843406646f38dba1d87601499f827"}
Sep 09 20:44:12.219141 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:12.219110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" event={"ID":"a1fc06bf-5cc6-4c31-9949-19e4333c2f94","Type":"ContainerStarted","Data":"dbca82855e827ece2617cab9a5206610f54ac62847095f7dd21d4e8ee812ba7a"}
Sep 09 20:44:12.243220 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:12.243162    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podStartSLOduration=3.10134833 podStartE2EDuration="12.243145153s" totalImagesPullingTime="7.956768927s" totalInitContainerRuntime="1.185027896s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:12.242400373 +0000 UTC m=+2960.831054032" watchObservedRunningTime="2026-09-09 20:44:12.243145153 +0000 UTC m=+2960.831798810"
Sep 09 20:44:13.132276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:13.132233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:18.560797 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:18.560712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:21.131983 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:21.131939    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:44:21.133515 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:21.133482    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:44:22.177002 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:22.176965    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:22.553001 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:22.552882    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:31.132964 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:31.132915    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:44:32.182404 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:32.182368    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:34.749345 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:34.749289    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:41.133090 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:41.133027    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:44:47.435799 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:47.435761    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:51.132208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:51.132162    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:44:52.078262 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:52.078233    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:44:52.088742 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:44:52.088712    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:45:01.132227 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:45:01.132186    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:45:11.132370 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:45:11.132268    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:45:21.132374 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:45:21.132299    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:45:31.132351 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:45:31.132294    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:45:41.132357 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:45:41.132299    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:45:51.132664 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:45:51.132614    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:46:01.132298 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:01.132251    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:46:11.142533 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:11.142488    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:46:11.150717 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:11.150688    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:46:17.051643 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:17.051606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:17.104479 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:17.104442    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"]
Sep 09 20:46:17.104801 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:17.104761    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="modelcar" containerID="cri-o://1b0528c4b6bdf204d338328b07602b16275843406646f38dba1d87601499f827" gracePeriod=60
Sep 09 20:46:17.351721 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:17.351683    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:17.803795 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:17.803712    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f_a1fc06bf-5cc6-4c31-9949-19e4333c2f94/modelcar/0.log"
Sep 09 20:46:17.804417 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:17.804390    2578 generic.go:356] "Generic (PLEG): container finished" podID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerID="1b0528c4b6bdf204d338328b07602b16275843406646f38dba1d87601499f827" exitCode=143
Sep 09 20:46:17.804674 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:17.804469    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" event={"ID":"a1fc06bf-5cc6-4c31-9949-19e4333c2f94","Type":"ContainerDied","Data":"1b0528c4b6bdf204d338328b07602b16275843406646f38dba1d87601499f827"}
Sep 09 20:46:21.522825 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:21.522769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:21.903504 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:21.903461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:32.128342 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:32.128264    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" containerID="cri-o://dbca82855e827ece2617cab9a5206610f54ac62847095f7dd21d4e8ee812ba7a" gracePeriod=45
Sep 09 20:46:32.873830 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:32.873799    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f_a1fc06bf-5cc6-4c31-9949-19e4333c2f94/modelcar/0.log"
Sep 09 20:46:32.874515 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:32.874490    2578 generic.go:356] "Generic (PLEG): container finished" podID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerID="dbca82855e827ece2617cab9a5206610f54ac62847095f7dd21d4e8ee812ba7a" exitCode=0
Sep 09 20:46:32.874623 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:32.874568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" event={"ID":"a1fc06bf-5cc6-4c31-9949-19e4333c2f94","Type":"ContainerDied","Data":"dbca82855e827ece2617cab9a5206610f54ac62847095f7dd21d4e8ee812ba7a"}
Sep 09 20:46:33.143905 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.143865    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 09 20:46:33.380140 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.380116    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f_a1fc06bf-5cc6-4c31-9949-19e4333c2f94/modelcar/0.log"
Sep 09 20:46:33.382846 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.382826    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:46:33.477981 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.477894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-dshm\") pod \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") "
Sep 09 20:46:33.477981 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.477948    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbxkk\" (UniqueName: \"kubernetes.io/projected/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kube-api-access-mbxkk\") pod \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") "
Sep 09 20:46:33.478180 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.477988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kserve-provision-location\") pod \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") "
Sep 09 20:46:33.478180 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.478030    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-model-cache\") pod \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") "
Sep 09 20:46:33.478180 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.478077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-home\") pod \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") "
Sep 09 20:46:33.478180 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.478141    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tls-certs\") pod \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") "
Sep 09 20:46:33.478180 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.478166    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tmp-dir\") pod \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\" (UID: \"a1fc06bf-5cc6-4c31-9949-19e4333c2f94\") "
Sep 09 20:46:33.478487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.478367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1fc06bf-5cc6-4c31-9949-19e4333c2f94" (UID: "a1fc06bf-5cc6-4c31-9949-19e4333c2f94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:33.478487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.478399    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1fc06bf-5cc6-4c31-9949-19e4333c2f94" (UID: "a1fc06bf-5cc6-4c31-9949-19e4333c2f94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:33.478487 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.478485    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:46:33.478670 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.478504    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:46:33.478769 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.478748    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-home" (OuterVolumeSpecName: "home") pod "a1fc06bf-5cc6-4c31-9949-19e4333c2f94" (UID: "a1fc06bf-5cc6-4c31-9949-19e4333c2f94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:33.480362 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.480328    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1fc06bf-5cc6-4c31-9949-19e4333c2f94" (UID: "a1fc06bf-5cc6-4c31-9949-19e4333c2f94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:33.480478 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.480417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-dshm" (OuterVolumeSpecName: "dshm") pod "a1fc06bf-5cc6-4c31-9949-19e4333c2f94" (UID: "a1fc06bf-5cc6-4c31-9949-19e4333c2f94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:33.480526 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.480500    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kube-api-access-mbxkk" (OuterVolumeSpecName: "kube-api-access-mbxkk") pod "a1fc06bf-5cc6-4c31-9949-19e4333c2f94" (UID: "a1fc06bf-5cc6-4c31-9949-19e4333c2f94"). InnerVolumeSpecName "kube-api-access-mbxkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:33.489754 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.489731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1fc06bf-5cc6-4c31-9949-19e4333c2f94" (UID: "a1fc06bf-5cc6-4c31-9949-19e4333c2f94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:33.579258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.579223    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:46:33.579258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.579252    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:46:33.579258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.579262    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:46:33.579492 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.579271    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:46:33.579492 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.579279    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbxkk\" (UniqueName: \"kubernetes.io/projected/a1fc06bf-5cc6-4c31-9949-19e4333c2f94-kube-api-access-mbxkk\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:46:33.880255 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.880229    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f_a1fc06bf-5cc6-4c31-9949-19e4333c2f94/modelcar/0.log"
Sep 09 20:46:33.881028 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.880998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f" event={"ID":"a1fc06bf-5cc6-4c31-9949-19e4333c2f94","Type":"ContainerDied","Data":"f9186c23551eb88b83a7b8e0c543ca8f6a791867622c44494415cbf21ca555cf"}
Sep 09 20:46:33.881153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.881068    2578 scope.go:117] "RemoveContainer" containerID="1b0528c4b6bdf204d338328b07602b16275843406646f38dba1d87601499f827"
Sep 09 20:46:33.881153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.881008    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"
Sep 09 20:46:33.889931 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.889904    2578 scope.go:117] "RemoveContainer" containerID="dbca82855e827ece2617cab9a5206610f54ac62847095f7dd21d4e8ee812ba7a"
Sep 09 20:46:33.898151 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.898133    2578 scope.go:117] "RemoveContainer" containerID="6999117f5b838002cbab8f9f0bf27e9425d18da16ce5d3c29b450a500305c70f"
Sep 09 20:46:33.911273 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.911238    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"]
Sep 09 20:46:33.921184 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:33.921162    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8fpvt8f"]
Sep 09 20:46:35.895266 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:46:35.895225    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" path="/var/lib/kubelet/pods/a1fc06bf-5cc6-4c31-9949-19e4333c2f94/volumes"
Sep 09 20:47:07.145058 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:07.145011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:08.988001 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:08.987963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:11.509610 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:11.509571    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:11.793966 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:11.793872    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:15.125345 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:15.125243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:15.671682 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:15.671642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:20.579970 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:20.579929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:22.031464 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:22.031426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:27.980913 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:27.980870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:28.162584 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:28.162547    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:28.429631 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:28.429598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:30.350837 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:30.350805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:30.585523 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:30.585488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:42.544677 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.544641    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"]
Sep 09 20:47:42.545121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.544961    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="modelcar"
Sep 09 20:47:42.545121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.544972    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="modelcar"
Sep 09 20:47:42.545121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.544990    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="modelcar-init"
Sep 09 20:47:42.545121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.544996    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="modelcar-init"
Sep 09 20:47:42.545121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.545009    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main"
Sep 09 20:47:42.545121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.545015    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main"
Sep 09 20:47:42.545121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.545066    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="modelcar"
Sep 09 20:47:42.545121 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.545073    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1fc06bf-5cc6-4c31-9949-19e4333c2f94" containerName="main"
Sep 09 20:47:42.547568 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.547548    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.550961 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.550932    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 09 20:47:42.551350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.550991    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-pxxb4\""
Sep 09 20:47:42.551788 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.551529    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 09 20:47:42.551788 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.551583    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 09 20:47:42.551788 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.550997    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:47:42.552051 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.552022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.553979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.552349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhsr\" (UniqueName: \"kubernetes.io/projected/33bbec50-4c34-4630-8d23-6cd31397bac5-kube-api-access-gvhsr\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.553979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.552507    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 09 20:47:42.553979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.552580    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.553979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.552752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.553979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.552816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.553979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.552889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.553979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.553050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bbec50-4c34-4630-8d23-6cd31397bac5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.553979 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.553172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33bbec50-4c34-4630-8d23-6cd31397bac5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.566924 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.566896    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"]
Sep 09 20:47:42.568362 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.568340    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:42.654062 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bbec50-4c34-4630-8d23-6cd31397bac5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33bbec50-4c34-4630-8d23-6cd31397bac5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhsr\" (UniqueName: \"kubernetes.io/projected/33bbec50-4c34-4630-8d23-6cd31397bac5-kube-api-access-gvhsr\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654276 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654640 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654579    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654708    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.654999 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.654975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33bbec50-4c34-4630-8d23-6cd31397bac5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.655072 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.655049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.656634 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.656615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.656927 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.656909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bbec50-4c34-4630-8d23-6cd31397bac5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.666967 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.666941    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhsr\" (UniqueName: \"kubernetes.io/projected/33bbec50-4c34-4630-8d23-6cd31397bac5-kube-api-access-gvhsr\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.826590 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.826511    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"]
Sep 09 20:47:42.831081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.831063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:47:42.836656 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.836632    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-56d8m\""
Sep 09 20:47:42.848991 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.848967    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"]
Sep 09 20:47:42.854174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.854152    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:42.856325 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.856292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd456df4-4f84-40d6-b5d1-31f5406867f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb\" (UID: \"bd456df4-4f84-40d6-b5d1-31f5406867f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:47:42.856402 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.856336    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnbt\" (UniqueName: \"kubernetes.io/projected/bd456df4-4f84-40d6-b5d1-31f5406867f6-kube-api-access-wrnbt\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb\" (UID: \"bd456df4-4f84-40d6-b5d1-31f5406867f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:47:42.861278 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.861260    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:42.956971 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.956927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd456df4-4f84-40d6-b5d1-31f5406867f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb\" (UID: \"bd456df4-4f84-40d6-b5d1-31f5406867f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:47:42.957160 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.956993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnbt\" (UniqueName: \"kubernetes.io/projected/bd456df4-4f84-40d6-b5d1-31f5406867f6-kube-api-access-wrnbt\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb\" (UID: \"bd456df4-4f84-40d6-b5d1-31f5406867f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:47:42.959578 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.959551    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd456df4-4f84-40d6-b5d1-31f5406867f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb\" (UID: \"bd456df4-4f84-40d6-b5d1-31f5406867f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:47:42.966743 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:42.966717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnbt\" (UniqueName: \"kubernetes.io/projected/bd456df4-4f84-40d6-b5d1-31f5406867f6-kube-api-access-wrnbt\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb\" (UID: \"bd456df4-4f84-40d6-b5d1-31f5406867f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:47:43.142022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:43.141984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:47:43.154468 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:43.154441    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"]
Sep 09 20:47:43.156252 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:47:43.156229    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bbec50_4c34_4630_8d23_6cd31397bac5.slice/crio-a10d11545b8391d582e73aef2ffba3e66be3df7c1de9505ebea2d74e84f1c49c WatchSource:0}: Error finding container a10d11545b8391d582e73aef2ffba3e66be3df7c1de9505ebea2d74e84f1c49c: Status 404 returned error can't find the container with id a10d11545b8391d582e73aef2ffba3e66be3df7c1de9505ebea2d74e84f1c49c
Sep 09 20:47:43.183150 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:43.182998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" event={"ID":"33bbec50-4c34-4630-8d23-6cd31397bac5","Type":"ContainerStarted","Data":"a10d11545b8391d582e73aef2ffba3e66be3df7c1de9505ebea2d74e84f1c49c"}
Sep 09 20:47:43.437894 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:47:43.437860    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd456df4_4f84_40d6_b5d1_31f5406867f6.slice/crio-b8f97d1295c83f1078f2d5a90c289569b07b6eb8b53320890f716d85002212ad WatchSource:0}: Error finding container b8f97d1295c83f1078f2d5a90c289569b07b6eb8b53320890f716d85002212ad: Status 404 returned error can't find the container with id b8f97d1295c83f1078f2d5a90c289569b07b6eb8b53320890f716d85002212ad
Sep 09 20:47:43.443378 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:43.443348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"]
Sep 09 20:47:44.189138 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:44.189101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" event={"ID":"33bbec50-4c34-4630-8d23-6cd31397bac5","Type":"ContainerStarted","Data":"ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7"}
Sep 09 20:47:44.190731 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:44.190697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" event={"ID":"bd456df4-4f84-40d6-b5d1-31f5406867f6","Type":"ContainerStarted","Data":"29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b"}
Sep 09 20:47:44.190839 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:44.190759    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:47:44.190839 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:44.190778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" event={"ID":"bd456df4-4f84-40d6-b5d1-31f5406867f6","Type":"ContainerStarted","Data":"b8f97d1295c83f1078f2d5a90c289569b07b6eb8b53320890f716d85002212ad"}
Sep 09 20:47:46.200775 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:46.200740    2578 generic.go:356] "Generic (PLEG): container finished" podID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerID="ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7" exitCode=0
Sep 09 20:47:46.201252 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:46.200820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" event={"ID":"33bbec50-4c34-4630-8d23-6cd31397bac5","Type":"ContainerDied","Data":"ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7"}
Sep 09 20:47:46.252726 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:46.252665    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" podStartSLOduration=4.252643841 podStartE2EDuration="4.252643841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:44.255606054 +0000 UTC m=+3172.844259711" watchObservedRunningTime="2026-09-09 20:47:46.252643841 +0000 UTC m=+3174.841297498"
Sep 09 20:47:47.207535 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:47.207492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" event={"ID":"33bbec50-4c34-4630-8d23-6cd31397bac5","Type":"ContainerStarted","Data":"d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31"}
Sep 09 20:47:47.309698 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:47.309633    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podStartSLOduration=2.5724600410000003 podStartE2EDuration="5.309609914s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.737149873s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:47.307395147 +0000 UTC m=+3175.896048804" watchObservedRunningTime="2026-09-09 20:47:47.309609914 +0000 UTC m=+3175.898263572"
Sep 09 20:47:47.862093 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:47.862039    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:52.862080 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:52.862041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:47:52.863418 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:52.863384    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 09 20:47:58.701193 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:47:58.701156    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:02.862400 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:48:02.862348    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 09 20:48:12.862054 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:48:12.862010    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 09 20:48:15.197420 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:48:15.197384    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:48:20.073698 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:48:20.073662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:22.861824 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:48:22.861778    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 09 20:48:32.862639 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:48:32.862593    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 09 20:48:42.862574 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:48:42.862518    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 09 20:48:52.862780 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:48:52.862729    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 09 20:49:02.872453 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:02.872422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:49:02.880336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:02.880292    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:49:05.190397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:05.190275    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"]
Sep 09 20:49:05.197520 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:05.197487    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:49:05.203632 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:05.203603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:05.210526 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:05.210495    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"]
Sep 09 20:49:05.301073 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:05.301037    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:15.197043 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:15.197004    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:49:20.211967 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.211894    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerName="main" containerID="cri-o://29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b" gracePeriod=45
Sep 09 20:49:20.232383 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.232348    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main" containerID="cri-o://d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31" gracePeriod=45
Sep 09 20:49:20.877924 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.877893    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:49:20.905100 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905067    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-dshm\") pod \"33bbec50-4c34-4630-8d23-6cd31397bac5\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") "
Sep 09 20:49:20.905252 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905108    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33bbec50-4c34-4630-8d23-6cd31397bac5-cabundle-cert\") pod \"33bbec50-4c34-4630-8d23-6cd31397bac5\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") "
Sep 09 20:49:20.905252 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bbec50-4c34-4630-8d23-6cd31397bac5-tls-certs\") pod \"33bbec50-4c34-4630-8d23-6cd31397bac5\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") "
Sep 09 20:49:20.905252 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-tmp-dir\") pod \"33bbec50-4c34-4630-8d23-6cd31397bac5\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") "
Sep 09 20:49:20.905252 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvhsr\" (UniqueName: \"kubernetes.io/projected/33bbec50-4c34-4630-8d23-6cd31397bac5-kube-api-access-gvhsr\") pod \"33bbec50-4c34-4630-8d23-6cd31397bac5\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") "
Sep 09 20:49:20.905252 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905249    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-kserve-provision-location\") pod \"33bbec50-4c34-4630-8d23-6cd31397bac5\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") "
Sep 09 20:49:20.906464 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905328    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-home\") pod \"33bbec50-4c34-4630-8d23-6cd31397bac5\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") "
Sep 09 20:49:20.906464 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-model-cache\") pod \"33bbec50-4c34-4630-8d23-6cd31397bac5\" (UID: \"33bbec50-4c34-4630-8d23-6cd31397bac5\") "
Sep 09 20:49:20.906464 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bbec50-4c34-4630-8d23-6cd31397bac5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33bbec50-4c34-4630-8d23-6cd31397bac5" (UID: "33bbec50-4c34-4630-8d23-6cd31397bac5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:20.906464 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905720    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33bbec50-4c34-4630-8d23-6cd31397bac5" (UID: "33bbec50-4c34-4630-8d23-6cd31397bac5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:20.906464 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905796    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-model-cache" (OuterVolumeSpecName: "model-cache") pod "33bbec50-4c34-4630-8d23-6cd31397bac5" (UID: "33bbec50-4c34-4630-8d23-6cd31397bac5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:20.906464 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.905913    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-home" (OuterVolumeSpecName: "home") pod "33bbec50-4c34-4630-8d23-6cd31397bac5" (UID: "33bbec50-4c34-4630-8d23-6cd31397bac5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:20.907808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.907769    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-dshm" (OuterVolumeSpecName: "dshm") pod "33bbec50-4c34-4630-8d23-6cd31397bac5" (UID: "33bbec50-4c34-4630-8d23-6cd31397bac5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:20.908037 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.908011    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bbec50-4c34-4630-8d23-6cd31397bac5-kube-api-access-gvhsr" (OuterVolumeSpecName: "kube-api-access-gvhsr") pod "33bbec50-4c34-4630-8d23-6cd31397bac5" (UID: "33bbec50-4c34-4630-8d23-6cd31397bac5"). InnerVolumeSpecName "kube-api-access-gvhsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:20.908154 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.908126    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bbec50-4c34-4630-8d23-6cd31397bac5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33bbec50-4c34-4630-8d23-6cd31397bac5" (UID: "33bbec50-4c34-4630-8d23-6cd31397bac5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:49:20.928494 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:20.928468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33bbec50-4c34-4630-8d23-6cd31397bac5" (UID: "33bbec50-4c34-4630-8d23-6cd31397bac5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:21.006694 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.006606    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:21.006694 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.006635    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33bbec50-4c34-4630-8d23-6cd31397bac5-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:21.006694 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.006646    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33bbec50-4c34-4630-8d23-6cd31397bac5-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:21.006694 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.006679    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:21.006694 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.006689    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhsr\" (UniqueName: \"kubernetes.io/projected/33bbec50-4c34-4630-8d23-6cd31397bac5-kube-api-access-gvhsr\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:21.006694 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.006698    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:21.006973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.006706    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:21.006973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.006714    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33bbec50-4c34-4630-8d23-6cd31397bac5-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:21.649178 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.649145    2578 generic.go:356] "Generic (PLEG): container finished" podID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerID="d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31" exitCode=0
Sep 09 20:49:21.649678 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.649194    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"
Sep 09 20:49:21.649678 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.649228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" event={"ID":"33bbec50-4c34-4630-8d23-6cd31397bac5","Type":"ContainerDied","Data":"d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31"}
Sep 09 20:49:21.649678 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.649290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" event={"ID":"33bbec50-4c34-4630-8d23-6cd31397bac5","Type":"ContainerDied","Data":"a10d11545b8391d582e73aef2ffba3e66be3df7c1de9505ebea2d74e84f1c49c"}
Sep 09 20:49:21.649678 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.649338    2578 scope.go:117] "RemoveContainer" containerID="d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31"
Sep 09 20:49:21.658571 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.658553    2578 scope.go:117] "RemoveContainer" containerID="ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7"
Sep 09 20:49:21.666362 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.666343    2578 scope.go:117] "RemoveContainer" containerID="d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31"
Sep 09 20:49:21.666658 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:49:21.666634    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31\": container with ID starting with d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31 not found: ID does not exist" containerID="d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31"
Sep 09 20:49:21.666764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.666672    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31"} err="failed to get container status \"d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31\": rpc error: code = NotFound desc = could not find container \"d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31\": container with ID starting with d219db6d34462bd9e160ad0d7cb20c6ed0a1604fca162871597047ebcd9cdc31 not found: ID does not exist"
Sep 09 20:49:21.666764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.666689    2578 scope.go:117] "RemoveContainer" containerID="ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7"
Sep 09 20:49:21.666968 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:49:21.666953    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7\": container with ID starting with ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7 not found: ID does not exist" containerID="ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7"
Sep 09 20:49:21.667018 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.666971    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7"} err="failed to get container status \"ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7\": rpc error: code = NotFound desc = could not find container \"ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7\": container with ID starting with ea2d4f2b835d9fee6df65693b6c8b73ab55ee8aaf79b1ab8b484fa06eedaeaa7 not found: ID does not exist"
Sep 09 20:49:21.672418 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.672394    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"]
Sep 09 20:49:21.677045 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.677024    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn"]
Sep 09 20:49:21.873850 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.873788    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667b8d499d-nm8bn" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Sep 09 20:49:21.897004 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:21.896958    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" path="/var/lib/kubelet/pods/33bbec50-4c34-4630-8d23-6cd31397bac5/volumes"
Sep 09 20:49:25.197414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:25.197370    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:49:25.197808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:25.197478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:49:30.820683 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:30.820642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:30.910458 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:30.910418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:35.197517 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:35.197473    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:49:45.196814 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:45.196773    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:49:50.455878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.455850    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:49:50.559142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.559057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd456df4-4f84-40d6-b5d1-31f5406867f6-tls-certs\") pod \"bd456df4-4f84-40d6-b5d1-31f5406867f6\" (UID: \"bd456df4-4f84-40d6-b5d1-31f5406867f6\") "
Sep 09 20:49:50.559280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.559168    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrnbt\" (UniqueName: \"kubernetes.io/projected/bd456df4-4f84-40d6-b5d1-31f5406867f6-kube-api-access-wrnbt\") pod \"bd456df4-4f84-40d6-b5d1-31f5406867f6\" (UID: \"bd456df4-4f84-40d6-b5d1-31f5406867f6\") "
Sep 09 20:49:50.561199 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.561167    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd456df4-4f84-40d6-b5d1-31f5406867f6-kube-api-access-wrnbt" (OuterVolumeSpecName: "kube-api-access-wrnbt") pod "bd456df4-4f84-40d6-b5d1-31f5406867f6" (UID: "bd456df4-4f84-40d6-b5d1-31f5406867f6"). InnerVolumeSpecName "kube-api-access-wrnbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:50.561337 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.561212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd456df4-4f84-40d6-b5d1-31f5406867f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd456df4-4f84-40d6-b5d1-31f5406867f6" (UID: "bd456df4-4f84-40d6-b5d1-31f5406867f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:49:50.659893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.659855    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd456df4-4f84-40d6-b5d1-31f5406867f6-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:50.659893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.659888    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrnbt\" (UniqueName: \"kubernetes.io/projected/bd456df4-4f84-40d6-b5d1-31f5406867f6-kube-api-access-wrnbt\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:49:50.785165 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.785128    2578 generic.go:356] "Generic (PLEG): container finished" podID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerID="29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b" exitCode=0
Sep 09 20:49:50.785371 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.785186    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"
Sep 09 20:49:50.785371 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.785224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" event={"ID":"bd456df4-4f84-40d6-b5d1-31f5406867f6","Type":"ContainerDied","Data":"29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b"}
Sep 09 20:49:50.785371 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.785337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb" event={"ID":"bd456df4-4f84-40d6-b5d1-31f5406867f6","Type":"ContainerDied","Data":"b8f97d1295c83f1078f2d5a90c289569b07b6eb8b53320890f716d85002212ad"}
Sep 09 20:49:50.785371 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.785369    2578 scope.go:117] "RemoveContainer" containerID="29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b"
Sep 09 20:49:50.794477 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.794448    2578 scope.go:117] "RemoveContainer" containerID="29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b"
Sep 09 20:49:50.794751 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:49:50.794732    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b\": container with ID starting with 29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b not found: ID does not exist" containerID="29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b"
Sep 09 20:49:50.794816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.794759    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b"} err="failed to get container status \"29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b\": rpc error: code = NotFound desc = could not find container \"29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b\": container with ID starting with 29237f0590cdc4beb4093b80fb1135e5e0230cc47b7240aaaec624d26adb759b not found: ID does not exist"
Sep 09 20:49:50.808240 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.808216    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"]
Sep 09 20:49:50.821920 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:50.821858    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schecc6fb"]
Sep 09 20:49:51.896808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:51.896772    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" path="/var/lib/kubelet/pods/bd456df4-4f84-40d6-b5d1-31f5406867f6/volumes"
Sep 09 20:49:52.103149 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:52.103116    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:49:52.115254 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:52.115230    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:49:53.514021 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:53.513981    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:53.605494 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:53.605455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:54.078866 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:54.078829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:54.178687 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:54.178645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:54.278366 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:54.278328    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:56.333005 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:56.332969    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:56.442877 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:56.442846    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:56.546356 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:49:56.546299    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:00.762549 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:00.762517    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:02.157205 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:02.157165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:02.258402 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:02.258369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:06.425627 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:06.425591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:06.486456 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:06.486420    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:07.655122 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.655089    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"]
Sep 09 20:50:07.655556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.655434    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="storage-initializer"
Sep 09 20:50:07.655556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.655447    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="storage-initializer"
Sep 09 20:50:07.655556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.655459    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main"
Sep 09 20:50:07.655556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.655464    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main"
Sep 09 20:50:07.655556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.655472    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerName="main"
Sep 09 20:50:07.655556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.655477    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerName="main"
Sep 09 20:50:07.655556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.655532    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd456df4-4f84-40d6-b5d1-31f5406867f6" containerName="main"
Sep 09 20:50:07.655556 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.655544    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="33bbec50-4c34-4630-8d23-6cd31397bac5" containerName="main"
Sep 09 20:50:07.658565 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.658543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.661570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.661520    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 09 20:50:07.661696 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.661578    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 09 20:50:07.661751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.661697    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Sep 09 20:50:07.661751 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.661719    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 09 20:50:07.661942 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.661925    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-xtqmb\""
Sep 09 20:50:07.672128 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.672107    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"]
Sep 09 20:50:07.682877 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.682856    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:07.802980 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.802948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-home\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.802980 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.802991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tpg5\" (UniqueName: \"kubernetes.io/projected/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kube-api-access-9tpg5\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.803197 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.803030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.803197 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.803053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.803197 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.803070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-dshm\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.803197 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.803098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.803197 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.803156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-model-cache\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904068 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-home\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904278 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tpg5\" (UniqueName: \"kubernetes.io/projected/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kube-api-access-9tpg5\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904278 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904278 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904278 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-dshm\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904278 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904278 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-model-cache\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904641 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-home\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904641 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904641 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-model-cache\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.904758 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.904664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.906534 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.906483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-dshm\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.907026 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.907004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.915794 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.915771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tpg5\" (UniqueName: \"kubernetes.io/projected/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kube-api-access-9tpg5\") pod \"e2e-del-blocked-svc-kserve-6c99569b8f-z5825\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:07.916197 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.916177    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:07.969203 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:07.969169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:08.244021 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:08.243912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"]
Sep 09 20:50:08.246342 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:50:08.246295    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8d796a_fb83_4ca3_99d0_d2cbfec2c207.slice/crio-d07a1088cf7802a256528c6f411439cbe6587bc53a7c9ca745a41cae5fd2d5ed WatchSource:0}: Error finding container d07a1088cf7802a256528c6f411439cbe6587bc53a7c9ca745a41cae5fd2d5ed: Status 404 returned error can't find the container with id d07a1088cf7802a256528c6f411439cbe6587bc53a7c9ca745a41cae5fd2d5ed
Sep 09 20:50:08.863860 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:08.863819    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825" event={"ID":"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207","Type":"ContainerStarted","Data":"43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e"}
Sep 09 20:50:08.864264 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:08.863876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825" event={"ID":"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207","Type":"ContainerStarted","Data":"d07a1088cf7802a256528c6f411439cbe6587bc53a7c9ca745a41cae5fd2d5ed"}
Sep 09 20:50:09.877500 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:09.877465    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"]
Sep 09 20:50:09.881600 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:09.881568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:09.932262 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:50:09.932227    2578 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 09 20:50:09.932463 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:50:09.932293    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs podName:0b8d796a-fb83-4ca3-99d0-d2cbfec2c207 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:10.432277209 +0000 UTC m=+3319.020930843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs") pod "e2e-del-blocked-svc-kserve-6c99569b8f-z5825" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 09 20:50:09.943757 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:09.943731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:10.437153 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:50:10.437113    2578 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 09 20:50:10.437344 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:50:10.437185    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs podName:0b8d796a-fb83-4ca3-99d0-d2cbfec2c207 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:11.437170088 +0000 UTC m=+3320.025823723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs") pod "e2e-del-blocked-svc-kserve-6c99569b8f-z5825" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 09 20:50:10.778490 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:10.778303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:10.876537 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:10.876470    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825" podUID="0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" containerName="storage-initializer" containerID="cri-o://43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e" gracePeriod=60
Sep 09 20:50:10.882411 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:10.882336    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:10.977793 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:10.977746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:11.445526 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:50:11.445488    2578 secret.go:189] Couldn't get secret e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-self-signed-certs: secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 09 20:50:11.445713 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:50:11.445563    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs podName:0b8d796a-fb83-4ca3-99d0-d2cbfec2c207 nodeName:}" failed. No retries permitted until 2026-09-09 20:50:13.445547368 +0000 UTC m=+3322.034201003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs") pod "e2e-del-blocked-svc-kserve-6c99569b8f-z5825" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207") : secret "e2e-del-blocked-svc-kserve-self-signed-certs" not found
Sep 09 20:50:11.581554 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:11.581520    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:13.032054 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.032027    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:13.058505 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-home\") pod \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") "
Sep 09 20:50:13.058699 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tpg5\" (UniqueName: \"kubernetes.io/projected/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kube-api-access-9tpg5\") pod \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") "
Sep 09 20:50:13.058699 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058568    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-dshm\") pod \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") "
Sep 09 20:50:13.058699 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058592    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kserve-provision-location\") pod \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") "
Sep 09 20:50:13.058699 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tmp-dir\") pod \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") "
Sep 09 20:50:13.058699 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058690    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-model-cache\") pod \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") "
Sep 09 20:50:13.058981 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs\") pod \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\" (UID: \"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207\") "
Sep 09 20:50:13.058981 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:13.059097 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:13.059156 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.058804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-home" (OuterVolumeSpecName: "home") pod "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:13.060898 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.060869    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-dshm" (OuterVolumeSpecName: "dshm") pod "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:13.061009 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.060942    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:13.061070 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.061041    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kube-api-access-9tpg5" (OuterVolumeSpecName: "kube-api-access-9tpg5") pod "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207"). InnerVolumeSpecName "kube-api-access-9tpg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:13.129994 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.129904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" (UID: "0b8d796a-fb83-4ca3-99d0-d2cbfec2c207"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:13.160336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.160287    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:50:13.160336 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.160335    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:50:13.160513 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.160345    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tpg5\" (UniqueName: \"kubernetes.io/projected/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kube-api-access-9tpg5\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:50:13.160513 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.160354    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:50:13.160513 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.160363    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:50:13.160513 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.160373    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:50:13.160513 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.160381    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:50:13.891764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.891727    2578 generic.go:356] "Generic (PLEG): container finished" podID="0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" containerID="43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e" exitCode=0
Sep 09 20:50:13.891932 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.891836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"
Sep 09 20:50:13.895766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.895740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825" event={"ID":"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207","Type":"ContainerDied","Data":"43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e"}
Sep 09 20:50:13.895924 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.895780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825" event={"ID":"0b8d796a-fb83-4ca3-99d0-d2cbfec2c207","Type":"ContainerDied","Data":"d07a1088cf7802a256528c6f411439cbe6587bc53a7c9ca745a41cae5fd2d5ed"}
Sep 09 20:50:13.895924 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.895805    2578 scope.go:117] "RemoveContainer" containerID="43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e"
Sep 09 20:50:13.905448 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.905422    2578 scope.go:117] "RemoveContainer" containerID="43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e"
Sep 09 20:50:13.905704 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:50:13.905682    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e\": container with ID starting with 43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e not found: ID does not exist" containerID="43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e"
Sep 09 20:50:13.905750 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.905714    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e"} err="failed to get container status \"43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e\": rpc error: code = NotFound desc = could not find container \"43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e\": container with ID starting with 43f5bb14cb9cc7b8a25e9329e970d3188c0bb73ca6367b017ec49b34a4c89c1e not found: ID does not exist"
Sep 09 20:50:13.933126 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.933100    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"]
Sep 09 20:50:13.937178 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:13.937154    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6c99569b8f-z5825"]
Sep 09 20:50:15.038448 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:15.038415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:15.143578 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:15.143542    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:15.239531 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:15.239491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:15.856112 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:15.856076    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:15.895806 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:15.895776    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" path="/var/lib/kubelet/pods/0b8d796a-fb83-4ca3-99d0-d2cbfec2c207/volumes"
Sep 09 20:50:19.443195 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:19.443165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:19.618683 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:19.618637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:19.723936 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:19.723840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:20.977503 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:20.977465    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:21.082492 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.082463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:21.179519 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.179483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:21.272627 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.272552    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"]
Sep 09 20:50:21.272864 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.272851    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" containerName="storage-initializer"
Sep 09 20:50:21.272920 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.272866    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" containerName="storage-initializer"
Sep 09 20:50:21.272972 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.272958    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8d796a-fb83-4ca3-99d0-d2cbfec2c207" containerName="storage-initializer"
Sep 09 20:50:21.277458 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.277436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.279888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.279863    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-sztc5\""
Sep 09 20:50:21.280176 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.280155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 09 20:50:21.280234 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.280196    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:50:21.280832 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.280809    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 09 20:50:21.286339 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.286301    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"]
Sep 09 20:50:21.305841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.305816    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:21.326930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.326903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.327033 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.326945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.327033 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.326970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.327033 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.326985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.327155 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.327044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.327155 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.327102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.327155 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.327127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.327275 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.327164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bxvj\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-kube-api-access-5bxvj\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.327275 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.327214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.383215 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.383179    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:21.428074 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428249 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428249 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428249 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428249 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428519 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428519 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bxvj\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-kube-api-access-5bxvj\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428519 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428519 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428713 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428713 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428713 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.428821 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.428750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.429127 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.429108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.431177 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.431145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.431371 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.431354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.440303 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.440277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.440545 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.440523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bxvj\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-kube-api-access-5bxvj\") pod \"router-gateway-1-openshift-default-6cc7987576-7w2d5\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.588387 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.588277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:21.866478 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.866444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"]
Sep 09 20:50:21.869721 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:50:21.869687    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f50b94_5ad8_4a5b_9127_9f5abb6bc2d8.slice/crio-fd219b4f5560db20e059fc9e76212b3bdd5886addfaa35da8d775ceafc7ca747 WatchSource:0}: Error finding container fd219b4f5560db20e059fc9e76212b3bdd5886addfaa35da8d775ceafc7ca747: Status 404 returned error can't find the container with id fd219b4f5560db20e059fc9e76212b3bdd5886addfaa35da8d775ceafc7ca747
Sep 09 20:50:21.871789 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.871749    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:50:21.871923 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.871816    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:50:21.871923 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.871844    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:50:21.931953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:21.931918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" event={"ID":"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8","Type":"ContainerStarted","Data":"fd219b4f5560db20e059fc9e76212b3bdd5886addfaa35da8d775ceafc7ca747"}
Sep 09 20:50:22.938282 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:22.938244    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" event={"ID":"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8","Type":"ContainerStarted","Data":"6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a"}
Sep 09 20:50:22.960816 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:22.960718    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" podStartSLOduration=1.960698738 podStartE2EDuration="1.960698738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:22.960030724 +0000 UTC m=+3331.548684382" watchObservedRunningTime="2026-09-09 20:50:22.960698738 +0000 UTC m=+3331.549352396"
Sep 09 20:50:23.589081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:23.589042    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:23.593796 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:23.593766    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:23.942359 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:23.942299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:23.943188 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:23.943167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:24.841020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:24.840993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:25.700428 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:25.700392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:25.802390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:25.802357    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:27.644566 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:27.644534    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:33.564631 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:33.564596    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:33.668552 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:33.668518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:34.653739 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.653703    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"]
Sep 09 20:50:34.656104 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.656085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:50:34.658810 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.658790    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-k2t9w\""
Sep 09 20:50:34.659746 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.659714    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Sep 09 20:50:34.659746 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.659733    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 09 20:50:34.659913 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.659825    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 09 20:50:34.671320 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.671287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"]
Sep 09 20:50:34.684474 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.684452    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:34.744836 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.744799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462dd637-fc5e-4622-8d7d-484c5a7e347c-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw\" (UID: \"462dd637-fc5e-4622-8d7d-484c5a7e347c\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:50:34.745011 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.744845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqcqk\" (UniqueName: \"kubernetes.io/projected/462dd637-fc5e-4622-8d7d-484c5a7e347c-kube-api-access-rqcqk\") pod \"e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw\" (UID: \"462dd637-fc5e-4622-8d7d-484c5a7e347c\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:50:34.846024 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.845984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462dd637-fc5e-4622-8d7d-484c5a7e347c-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw\" (UID: \"462dd637-fc5e-4622-8d7d-484c5a7e347c\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:50:34.846210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.846044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcqk\" (UniqueName: \"kubernetes.io/projected/462dd637-fc5e-4622-8d7d-484c5a7e347c-kube-api-access-rqcqk\") pod \"e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw\" (UID: \"462dd637-fc5e-4622-8d7d-484c5a7e347c\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:50:34.846397 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.846378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462dd637-fc5e-4622-8d7d-484c5a7e347c-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw\" (UID: \"462dd637-fc5e-4622-8d7d-484c5a7e347c\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:50:34.855349 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.855298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcqk\" (UniqueName: \"kubernetes.io/projected/462dd637-fc5e-4622-8d7d-484c5a7e347c-kube-api-access-rqcqk\") pod \"e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw\" (UID: \"462dd637-fc5e-4622-8d7d-484c5a7e347c\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:50:34.966717 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.966617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:50:34.971206 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:34.971183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:35.004364 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:35.004216    2578 generic.go:356] "Generic (PLEG): container finished" podID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerID="6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a" exitCode=0
Sep 09 20:50:35.004364 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:35.004290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" event={"ID":"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8","Type":"ContainerDied","Data":"6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a"}
Sep 09 20:50:35.004841 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:35.004738    2578 scope.go:117] "RemoveContainer" containerID="6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a"
Sep 09 20:50:35.005895 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:35.005863    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:50:35.006006 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:35.005941    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:50:35.006006 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:35.005979    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 20:50:35.243745 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:35.243540    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"]
Sep 09 20:50:35.246574 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:50:35.246545    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462dd637_fc5e_4622_8d7d_484c5a7e347c.slice/crio-a943509447c9f9c99f00e03a2e77d9b92853ec4485582f8007ad9b4346c2770f WatchSource:0}: Error finding container a943509447c9f9c99f00e03a2e77d9b92853ec4485582f8007ad9b4346c2770f: Status 404 returned error can't find the container with id a943509447c9f9c99f00e03a2e77d9b92853ec4485582f8007ad9b4346c2770f
Sep 09 20:50:36.010803 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:36.010749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" event={"ID":"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8","Type":"ContainerStarted","Data":"c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d"}
Sep 09 20:50:36.012570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:36.012540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw" event={"ID":"462dd637-fc5e-4622-8d7d-484c5a7e347c","Type":"ContainerStarted","Data":"6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6"}
Sep 09 20:50:36.012708 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:36.012593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw" event={"ID":"462dd637-fc5e-4622-8d7d-484c5a7e347c","Type":"ContainerStarted","Data":"a943509447c9f9c99f00e03a2e77d9b92853ec4485582f8007ad9b4346c2770f"}
Sep 09 20:50:36.588949 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:36.588899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:36.590040 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:36.590004    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 20:50:37.588947 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:37.588903    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 20:50:38.589103 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:38.589055    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 20:50:38.942645 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:38.942605    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:39.588875 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:39.588824    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 20:50:40.589547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:40.589505    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 09 20:50:41.036655 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:41.036619    2578 generic.go:356] "Generic (PLEG): container finished" podID="462dd637-fc5e-4622-8d7d-484c5a7e347c" containerID="6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6" exitCode=0
Sep 09 20:50:41.036826 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:41.036691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw" event={"ID":"462dd637-fc5e-4622-8d7d-484c5a7e347c","Type":"ContainerDied","Data":"6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6"}
Sep 09 20:50:41.593676 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:41.593638    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:41.594796 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:41.594775    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:50:42.043021 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:42.042926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw" event={"ID":"462dd637-fc5e-4622-8d7d-484c5a7e347c","Type":"ContainerStarted","Data":"352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62"}
Sep 09 20:50:42.062197 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:42.062142    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw" podStartSLOduration=2.410468689 podStartE2EDuration="8.062126966s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.651658277s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:42.060517476 +0000 UTC m=+3350.649171132" watchObservedRunningTime="2026-09-09 20:50:42.062126966 +0000 UTC m=+3350.650780623"
Sep 09 20:50:49.308400 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:49.308364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:49.820363 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:49.820302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:49.925225 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:49.925193    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:51.261338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:51.261286    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:51.261916 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:51.261894    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"]
Sep 09 20:50:51.262090 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:51.262072    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw" podUID="462dd637-fc5e-4622-8d7d-484c5a7e347c" containerName="main" containerID="cri-o://352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62" gracePeriod=30
Sep 09 20:50:51.347761 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:51.347721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:54.302597 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:54.302562    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:54.394791 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:54.394758    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:59.436499 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:50:59.436449    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:00.516132 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:00.516102    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:51:00.585531 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:00.585456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqcqk\" (UniqueName: \"kubernetes.io/projected/462dd637-fc5e-4622-8d7d-484c5a7e347c-kube-api-access-rqcqk\") pod \"462dd637-fc5e-4622-8d7d-484c5a7e347c\" (UID: \"462dd637-fc5e-4622-8d7d-484c5a7e347c\") "
Sep 09 20:51:00.585531 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:00.585495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462dd637-fc5e-4622-8d7d-484c5a7e347c-kserve-provision-location\") pod \"462dd637-fc5e-4622-8d7d-484c5a7e347c\" (UID: \"462dd637-fc5e-4622-8d7d-484c5a7e347c\") "
Sep 09 20:51:00.587681 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:00.587648    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462dd637-fc5e-4622-8d7d-484c5a7e347c-kube-api-access-rqcqk" (OuterVolumeSpecName: "kube-api-access-rqcqk") pod "462dd637-fc5e-4622-8d7d-484c5a7e347c" (UID: "462dd637-fc5e-4622-8d7d-484c5a7e347c"). InnerVolumeSpecName "kube-api-access-rqcqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:51:00.643592 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:00.643550    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/462dd637-fc5e-4622-8d7d-484c5a7e347c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "462dd637-fc5e-4622-8d7d-484c5a7e347c" (UID: "462dd637-fc5e-4622-8d7d-484c5a7e347c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:51:00.686817 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:00.686788    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqcqk\" (UniqueName: \"kubernetes.io/projected/462dd637-fc5e-4622-8d7d-484c5a7e347c-kube-api-access-rqcqk\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:51:00.686817 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:00.686816    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/462dd637-fc5e-4622-8d7d-484c5a7e347c-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:51:01.133865 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.133830    2578 generic.go:356] "Generic (PLEG): container finished" podID="462dd637-fc5e-4622-8d7d-484c5a7e347c" containerID="352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62" exitCode=1
Sep 09 20:51:01.134053 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.133878    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw" event={"ID":"462dd637-fc5e-4622-8d7d-484c5a7e347c","Type":"ContainerDied","Data":"352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62"}
Sep 09 20:51:01.134053 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.133890    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"
Sep 09 20:51:01.134053 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.133918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw" event={"ID":"462dd637-fc5e-4622-8d7d-484c5a7e347c","Type":"ContainerDied","Data":"a943509447c9f9c99f00e03a2e77d9b92853ec4485582f8007ad9b4346c2770f"}
Sep 09 20:51:01.134053 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.133941    2578 scope.go:117] "RemoveContainer" containerID="352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62"
Sep 09 20:51:01.142404 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.142220    2578 scope.go:117] "RemoveContainer" containerID="6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6"
Sep 09 20:51:01.150169 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.150151    2578 scope.go:117] "RemoveContainer" containerID="352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62"
Sep 09 20:51:01.150474 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:51:01.150454    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62\": container with ID starting with 352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62 not found: ID does not exist" containerID="352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62"
Sep 09 20:51:01.150529 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.150484    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62"} err="failed to get container status \"352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62\": rpc error: code = NotFound desc = could not find container \"352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62\": container with ID starting with 352e1a71c6106308fab9849dcd9a510f8c7298b2b8fe70bc42df99dc11d80f62 not found: ID does not exist"
Sep 09 20:51:01.150529 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.150502    2578 scope.go:117] "RemoveContainer" containerID="6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6"
Sep 09 20:51:01.150734 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:51:01.150710    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6\": container with ID starting with 6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6 not found: ID does not exist" containerID="6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6"
Sep 09 20:51:01.150776 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.150736    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6"} err="failed to get container status \"6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6\": rpc error: code = NotFound desc = could not find container \"6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6\": container with ID starting with 6beb32023e8166e0c86728be38d13eed6639447e57b8bbb6f00527d76b1874b6 not found: ID does not exist"
Sep 09 20:51:01.157331 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.157297    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"]
Sep 09 20:51:01.159758 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.159738    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-77697f8566-x8qcw"]
Sep 09 20:51:01.895791 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:01.895755    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462dd637-fc5e-4622-8d7d-484c5a7e347c" path="/var/lib/kubelet/pods/462dd637-fc5e-4622-8d7d-484c5a7e347c/volumes"
Sep 09 20:51:07.729022 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:07.728988    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:15.950214 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.950179    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"]
Sep 09 20:51:15.950623 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.950504    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462dd637-fc5e-4622-8d7d-484c5a7e347c" containerName="storage-initializer"
Sep 09 20:51:15.950623 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.950516    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="462dd637-fc5e-4622-8d7d-484c5a7e347c" containerName="storage-initializer"
Sep 09 20:51:15.950623 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.950529    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462dd637-fc5e-4622-8d7d-484c5a7e347c" containerName="main"
Sep 09 20:51:15.950623 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.950534    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="462dd637-fc5e-4622-8d7d-484c5a7e347c" containerName="main"
Sep 09 20:51:15.950623 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.950592    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="462dd637-fc5e-4622-8d7d-484c5a7e347c" containerName="main"
Sep 09 20:51:15.952714 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.952698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:15.955155 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.955125    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 09 20:51:15.955848 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.955827    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:51:15.955986 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.955900    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 09 20:51:15.956086 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.956068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-6htds\""
Sep 09 20:51:15.956141 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.956114    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 09 20:51:15.956338 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.956322    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 09 20:51:15.967198 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.967174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"]
Sep 09 20:51:15.989440 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:15.989408    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:16.014716 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.014686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-cabundle-cert\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.014855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.014732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-model-cache\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.014855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.014769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mhz\" (UniqueName: \"kubernetes.io/projected/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kube-api-access-x6mhz\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.014855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.014809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tmp-dir\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.014986 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.014859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kserve-provision-location\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.014986 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.014892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-home\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.014986 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.014923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-dshm\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.014986 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.014968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tls-certs\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.115777 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.115747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-model-cache\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.115777 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.115779    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mhz\" (UniqueName: \"kubernetes.io/projected/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kube-api-access-x6mhz\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.115798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tmp-dir\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.115830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kserve-provision-location\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.115863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-home\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.115879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-dshm\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.115906    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tls-certs\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116039 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.115953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-cabundle-cert\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116395 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.116245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-model-cache\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116395 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.116281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tmp-dir\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116395 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.116345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-home\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116545 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.116453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kserve-provision-location\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.116658 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.116639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-cabundle-cert\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.118354 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.118337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-dshm\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.118505 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.118487    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tls-certs\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.124470 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.124443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mhz\" (UniqueName: \"kubernetes.io/projected/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kube-api-access-x6mhz\") pod \"stop-feature-test-kserve-cd46bf7bc-h86qx\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.237377 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.237268    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:16.266011 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.265986    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:16.547567 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:16.547542    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"]
Sep 09 20:51:16.550082 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:51:16.550059    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde2ea31_6b9b_4d1b_9a14_c5b07b98a944.slice/crio-a4ad2cb53eb3acdf56477b2f287eda1efc42e1734f82e89debe8cc2c2e9c0ffd WatchSource:0}: Error finding container a4ad2cb53eb3acdf56477b2f287eda1efc42e1734f82e89debe8cc2c2e9c0ffd: Status 404 returned error can't find the container with id a4ad2cb53eb3acdf56477b2f287eda1efc42e1734f82e89debe8cc2c2e9c0ffd
Sep 09 20:51:17.209318 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:17.209264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" event={"ID":"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944","Type":"ContainerStarted","Data":"e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05"}
Sep 09 20:51:17.209727 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:17.209347    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" event={"ID":"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944","Type":"ContainerStarted","Data":"a4ad2cb53eb3acdf56477b2f287eda1efc42e1734f82e89debe8cc2c2e9c0ffd"}
Sep 09 20:51:19.221112 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:19.221072    2578 generic.go:356] "Generic (PLEG): container finished" podID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerID="e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05" exitCode=0
Sep 09 20:51:19.221565 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:19.221140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" event={"ID":"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944","Type":"ContainerDied","Data":"e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05"}
Sep 09 20:51:20.227967 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:20.227931    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" event={"ID":"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944","Type":"ContainerStarted","Data":"b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f"}
Sep 09 20:51:20.254923 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:20.254848    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podStartSLOduration=3.033323933 podStartE2EDuration="5.254832659s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.221508726s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:20.252551939 +0000 UTC m=+3388.841205596" watchObservedRunningTime="2026-09-09 20:51:20.254832659 +0000 UTC m=+3388.843486316"
Sep 09 20:51:20.266554 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:20.266526    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:26.266735 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:26.266703    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:51:26.268091 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:26.268049    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:51:32.750958 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:32.750924    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:36.266902 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:36.266850    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:51:46.267258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:46.267149    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:51:56.267357 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:51:56.267281    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:52:06.266873 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:52:06.266812    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:52:07.799883 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:52:07.799840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:16.266878 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:52:16.266837    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:52:26.267044 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:52:26.267002    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:52:36.267590 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:52:36.267536    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:52:46.266678 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:52:46.266634    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:52:56.266814 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:52:56.266768    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:53:04.710280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:04.710237    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:06.266660 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:06.266604    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 09 20:53:06.704739 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:06.704703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:07.732704 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:07.732664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:16.276668 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:16.276581    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:53:16.284888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:16.284860    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:53:18.387987 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:18.387952    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"]
Sep 09 20:53:18.569727 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:18.569681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:18.717473 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:18.717377    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:18.829504 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:18.829472    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:33.410613 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:33.410573    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main" containerID="cri-o://b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f" gracePeriod=45
Sep 09 20:53:34.161599 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.161574    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:53:34.216390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.212925    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kserve-provision-location\") pod \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") "
Sep 09 20:53:34.216390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.212986    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tmp-dir\") pod \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") "
Sep 09 20:53:34.216390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.213032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-cabundle-cert\") pod \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") "
Sep 09 20:53:34.216390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.213086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-dshm\") pod \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") "
Sep 09 20:53:34.216390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.213136    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-home\") pod \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") "
Sep 09 20:53:34.216390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.213175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-model-cache\") pod \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") "
Sep 09 20:53:34.216390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.213209    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tls-certs\") pod \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") "
Sep 09 20:53:34.216390 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.213256    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6mhz\" (UniqueName: \"kubernetes.io/projected/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kube-api-access-x6mhz\") pod \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\" (UID: \"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944\") "
Sep 09 20:53:34.216896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.216531    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kube-api-access-x6mhz" (OuterVolumeSpecName: "kube-api-access-x6mhz") pod "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" (UID: "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944"). InnerVolumeSpecName "kube-api-access-x6mhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:34.218671 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.218610    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-model-cache" (OuterVolumeSpecName: "model-cache") pod "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" (UID: "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:34.218868 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.218786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" (UID: "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:53:34.219092 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.219051    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-home" (OuterVolumeSpecName: "home") pod "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" (UID: "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:34.221075 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.220875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-dshm" (OuterVolumeSpecName: "dshm") pod "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" (UID: "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:34.221371 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.221345    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" (UID: "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:34.231570 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.231534    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" (UID: "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:34.240272 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.240248    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" (UID: "cde2ea31-6b9b-4d1b-9a14-c5b07b98a944"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:53:34.314918 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.314882    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:53:34.314918 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.314913    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:53:34.314918 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.314923    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:53:34.315142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.314933    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6mhz\" (UniqueName: \"kubernetes.io/projected/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kube-api-access-x6mhz\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:53:34.315142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.314943    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:53:34.315142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.314951    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:53:34.315142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.314959    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:53:34.315142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.314967    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:53:34.867955 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.867913    2578 generic.go:356] "Generic (PLEG): container finished" podID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerID="b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f" exitCode=0
Sep 09 20:53:34.868499 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.867952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" event={"ID":"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944","Type":"ContainerDied","Data":"b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f"}
Sep 09 20:53:34.868499 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.867976    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"
Sep 09 20:53:34.868499 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.868035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx" event={"ID":"cde2ea31-6b9b-4d1b-9a14-c5b07b98a944","Type":"ContainerDied","Data":"a4ad2cb53eb3acdf56477b2f287eda1efc42e1734f82e89debe8cc2c2e9c0ffd"}
Sep 09 20:53:34.868499 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.868059    2578 scope.go:117] "RemoveContainer" containerID="b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f"
Sep 09 20:53:34.877362 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.877345    2578 scope.go:117] "RemoveContainer" containerID="e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05"
Sep 09 20:53:34.884653 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.884629    2578 scope.go:117] "RemoveContainer" containerID="b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f"
Sep 09 20:53:34.884911 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:53:34.884893    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f\": container with ID starting with b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f not found: ID does not exist" containerID="b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f"
Sep 09 20:53:34.884961 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.884918    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f"} err="failed to get container status \"b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f\": rpc error: code = NotFound desc = could not find container \"b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f\": container with ID starting with b0a37ce2487a96f542e4d84257fa5843f6db2e29b0d02d6c508077d3d8f6ef0f not found: ID does not exist"
Sep 09 20:53:34.884961 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.884934    2578 scope.go:117] "RemoveContainer" containerID="e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05"
Sep 09 20:53:34.885227 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:53:34.885205    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05\": container with ID starting with e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05 not found: ID does not exist" containerID="e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05"
Sep 09 20:53:34.885284 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.885238    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05"} err="failed to get container status \"e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05\": rpc error: code = NotFound desc = could not find container \"e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05\": container with ID starting with e2de55e9efc54c11b67755f37fc5f8d1cbf77580f73e0501d090b9ec62bc7c05 not found: ID does not exist"
Sep 09 20:53:34.900244 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.900222    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"]
Sep 09 20:53:34.907914 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:34.907888    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-h86qx"]
Sep 09 20:53:35.896153 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:35.896121    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" path="/var/lib/kubelet/pods/cde2ea31-6b9b-4d1b-9a14-c5b07b98a944/volumes"
Sep 09 20:53:41.971001 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.970966    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"]
Sep 09 20:53:41.971507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.971268    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="storage-initializer"
Sep 09 20:53:41.971507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.971279    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="storage-initializer"
Sep 09 20:53:41.971507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.971287    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main"
Sep 09 20:53:41.971507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.971292    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main"
Sep 09 20:53:41.971507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.971386    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde2ea31-6b9b-4d1b-9a14-c5b07b98a944" containerName="main"
Sep 09 20:53:41.973350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.973330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:41.975895 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.975871    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 09 20:53:41.976447 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.976425    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 09 20:53:41.976593 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.976501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 09 20:53:41.976593 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.976476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:53:41.976732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.976507    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-6htds\""
Sep 09 20:53:41.976732 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.976425    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 09 20:53:41.990985 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.990963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:41.994847 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:41.994822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"]
Sep 09 20:53:42.077122 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.077081    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-model-cache\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.077360 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.077143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tls-certs\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.077360 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.077176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bblv\" (UniqueName: \"kubernetes.io/projected/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kube-api-access-5bblv\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.077360 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.077218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-home\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.077360 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.077259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tmp-dir\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.077360 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.077341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-cabundle-cert\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.077589 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.077384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kserve-provision-location\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.077589 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.077436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-dshm\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.178461 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.178425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tls-certs\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.178616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.178467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bblv\" (UniqueName: \"kubernetes.io/projected/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kube-api-access-5bblv\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.178616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.178500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-home\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.178616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.178530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tmp-dir\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.178616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.178554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-cabundle-cert\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.178616 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.178583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kserve-provision-location\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.178906 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.178619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-dshm\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.178906 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.178644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-model-cache\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.179048 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.179021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-home\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.179238 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.179060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tmp-dir\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.179342 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.179094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kserve-provision-location\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.179342 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.179130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-model-cache\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.179462 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.179442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-cabundle-cert\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.180982 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.180961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-dshm\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.181238 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.181220    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tls-certs\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.191237 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.191209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bblv\" (UniqueName: \"kubernetes.io/projected/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kube-api-access-5bblv\") pod \"stop-feature-test-kserve-cd46bf7bc-nx9mw\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.283525 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.283434    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:42.388840 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.388806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:42.590763 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.590734    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"]
Sep 09 20:53:42.597232 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:53:42.597197    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9dd516_d1fd_4e99_99a1_0238e38dacc9.slice/crio-171a0bfdb1de06251be0757db66817dc63024757275e19fd635625d9f3c620e4 WatchSource:0}: Error finding container 171a0bfdb1de06251be0757db66817dc63024757275e19fd635625d9f3c620e4: Status 404 returned error can't find the container with id 171a0bfdb1de06251be0757db66817dc63024757275e19fd635625d9f3c620e4
Sep 09 20:53:42.905658 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.905562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" event={"ID":"7b9dd516-d1fd-4e99-99a1-0238e38dacc9","Type":"ContainerStarted","Data":"0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec"}
Sep 09 20:53:42.905658 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:42.905625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" event={"ID":"7b9dd516-d1fd-4e99-99a1-0238e38dacc9","Type":"ContainerStarted","Data":"171a0bfdb1de06251be0757db66817dc63024757275e19fd635625d9f3c620e4"}
Sep 09 20:53:44.915647 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:44.915611    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerID="0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec" exitCode=0
Sep 09 20:53:44.916020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:44.915654    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" event={"ID":"7b9dd516-d1fd-4e99-99a1-0238e38dacc9","Type":"ContainerDied","Data":"0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec"}
Sep 09 20:53:45.921610 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:45.921575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" event={"ID":"7b9dd516-d1fd-4e99-99a1-0238e38dacc9","Type":"ContainerStarted","Data":"952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1"}
Sep 09 20:53:45.964661 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:45.964609    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podStartSLOduration=2.8011177050000002 podStartE2EDuration="4.964591041s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.163473336s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:45.963855709 +0000 UTC m=+3534.552509366" watchObservedRunningTime="2026-09-09 20:53:45.964591041 +0000 UTC m=+3534.553244697"
Sep 09 20:53:46.284448 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:46.284358    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:52.284502 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:52.284457    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:53:52.286103 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:52.286076    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:53:52.824860 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:52.824806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:54.837669 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:54.837632    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:55.207419 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:55.207383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:59.740255 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:59.740215    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"]
Sep 09 20:53:59.740713 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:59.740474    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy" containerID="cri-o://c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d" gracePeriod=30
Sep 09 20:53:59.919058 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:53:59.919019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:02.284209 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:02.284164    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:54:04.994918 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:04.994891    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:54:05.013738 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.013644    2578 generic.go:356] "Generic (PLEG): container finished" podID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerID="c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d" exitCode=0
Sep 09 20:54:05.013738 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.013693    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"
Sep 09 20:54:05.013738 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.013719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" event={"ID":"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8","Type":"ContainerDied","Data":"c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d"}
Sep 09 20:54:05.013984 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.013771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5" event={"ID":"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8","Type":"ContainerDied","Data":"fd219b4f5560db20e059fc9e76212b3bdd5886addfaa35da8d775ceafc7ca747"}
Sep 09 20:54:05.013984 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.013793    2578 scope.go:117] "RemoveContainer" containerID="c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d"
Sep 09 20:54:05.022425 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.022394    2578 scope.go:117] "RemoveContainer" containerID="6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a"
Sep 09 20:54:05.032529 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.032506    2578 scope.go:117] "RemoveContainer" containerID="c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d"
Sep 09 20:54:05.032830 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:54:05.032812    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d\": container with ID starting with c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d not found: ID does not exist" containerID="c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d"
Sep 09 20:54:05.032916 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.032847    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d"} err="failed to get container status \"c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d\": rpc error: code = NotFound desc = could not find container \"c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d\": container with ID starting with c1d4daa6f10ac81757aedeb399371e489ea2408d0e83ec0136a622833b39d58d not found: ID does not exist"
Sep 09 20:54:05.032916 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.032866    2578 scope.go:117] "RemoveContainer" containerID="6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a"
Sep 09 20:54:05.033159 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:54:05.033142    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a\": container with ID starting with 6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a not found: ID does not exist" containerID="6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a"
Sep 09 20:54:05.033204 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.033163    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a"} err="failed to get container status \"6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a\": rpc error: code = NotFound desc = could not find container \"6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a\": container with ID starting with 6c57b3f55394bdf22b7c036f587882cd61e21b24c17e9004c4be8d673db7488a not found: ID does not exist"
Sep 09 20:54:05.082345 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-data\") pod \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") "
Sep 09 20:54:05.082507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082387    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-credential-socket\") pod \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") "
Sep 09 20:54:05.082507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082420    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-token\") pod \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") "
Sep 09 20:54:05.082507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-podinfo\") pod \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") "
Sep 09 20:54:05.082507 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082495    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bxvj\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-kube-api-access-5bxvj\") pod \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") "
Sep 09 20:54:05.082758 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082522    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-socket\") pod \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") "
Sep 09 20:54:05.082758 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082573    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-envoy\") pod \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") "
Sep 09 20:54:05.082758 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082589    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" (UID: "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:05.082758 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istiod-ca-cert\") pod \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") "
Sep 09 20:54:05.082758 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082651    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" (UID: "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:05.082758 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-certs\") pod \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\" (UID: \"63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8\") "
Sep 09 20:54:05.083081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082820    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" (UID: "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:05.083081 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.082943    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" (UID: "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:05.083176 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.083076    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:54:05.083176 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.083099    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istiod-ca-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:54:05.083176 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.083114    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-data\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:54:05.083176 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.083127    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-credential-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:54:05.083388 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.083223    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" (UID: "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:05.085433 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.085402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-kube-api-access-5bxvj" (OuterVolumeSpecName: "kube-api-access-5bxvj") pod "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" (UID: "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8"). InnerVolumeSpecName "kube-api-access-5bxvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:05.085572 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.085544    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" (UID: "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 20:54:05.085901 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.085878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" (UID: "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:05.086450 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.086433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" (UID: "63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:05.184486 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.184444    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-podinfo\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:54:05.184486 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.184476    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bxvj\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-kube-api-access-5bxvj\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:54:05.184486 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.184486    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-envoy\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:54:05.184486 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.184494    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-workload-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:54:05.184754 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.184502    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8-istio-token\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:54:05.344855 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.344819    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"]
Sep 09 20:54:05.357631 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.357604    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-7w2d5"]
Sep 09 20:54:05.897976 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:05.897939    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" path="/var/lib/kubelet/pods/63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8/volumes"
Sep 09 20:54:12.121727 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.121687    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"]
Sep 09 20:54:12.122174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.122016    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy"
Sep 09 20:54:12.122174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.122030    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy"
Sep 09 20:54:12.122174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.122039    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy"
Sep 09 20:54:12.122174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.122044    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy"
Sep 09 20:54:12.122174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.122119    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy"
Sep 09 20:54:12.122174 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.122128    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f50b94-5ad8-4a5b-9127-9f5abb6bc2d8" containerName="istio-proxy"
Sep 09 20:54:12.125172 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.125153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.128596 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.128570    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 09 20:54:12.128930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.128907    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 09 20:54:12.129024 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.129003    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 09 20:54:12.129899 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.129875    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-zrrxw\""
Sep 09 20:54:12.130013 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.129914    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 09 20:54:12.131819 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.131801    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:54:12.156555 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.156524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"]
Sep 09 20:54:12.194178 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.194140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:12.249924 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.249885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.249924 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.249927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.250164 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.249988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gg2c\" (UniqueName: \"kubernetes.io/projected/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kube-api-access-5gg2c\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.250164 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.250037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.250164 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.250066    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.250164 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.250083    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.250350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.250194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.250350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.250232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.284438 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.284383    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:54:12.351245 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351245 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gg2c\" (UniqueName: \"kubernetes.io/projected/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kube-api-access-5gg2c\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351839 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351839 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351788    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.351923 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.351857    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.352058 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.352031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.352209 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.352187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.353833 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.353813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.354292 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.354268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.360096 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.360073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gg2c\" (UniqueName: \"kubernetes.io/projected/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kube-api-access-5gg2c\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.436280 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.436182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:12.721258 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.721225    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"]
Sep 09 20:54:12.723298 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:54:12.723269    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd453f43a_682f_444a_bd4a_0cba8b3ea8b2.slice/crio-c2a4bb88c1844b74570ae23f847a8b47475fba42eb97b8493e3c65b1e31472c3 WatchSource:0}: Error finding container c2a4bb88c1844b74570ae23f847a8b47475fba42eb97b8493e3c65b1e31472c3: Status 404 returned error can't find the container with id c2a4bb88c1844b74570ae23f847a8b47475fba42eb97b8493e3c65b1e31472c3
Sep 09 20:54:12.739380 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:12.739348    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:13.054888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:13.054786    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" event={"ID":"d453f43a-682f-444a-bd4a-0cba8b3ea8b2","Type":"ContainerStarted","Data":"a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369"}
Sep 09 20:54:13.054888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:13.054844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" event={"ID":"d453f43a-682f-444a-bd4a-0cba8b3ea8b2","Type":"ContainerStarted","Data":"c2a4bb88c1844b74570ae23f847a8b47475fba42eb97b8493e3c65b1e31472c3"}
Sep 09 20:54:16.072248 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:16.072206    2578 generic.go:356] "Generic (PLEG): container finished" podID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerID="a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369" exitCode=0
Sep 09 20:54:16.072785 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:16.072277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" event={"ID":"d453f43a-682f-444a-bd4a-0cba8b3ea8b2","Type":"ContainerDied","Data":"a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369"}
Sep 09 20:54:17.079051 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:17.079018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" event={"ID":"d453f43a-682f-444a-bd4a-0cba8b3ea8b2","Type":"ContainerStarted","Data":"c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660"}
Sep 09 20:54:17.107665 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:17.107602    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podStartSLOduration=2.656058088 podStartE2EDuration="5.107581095s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.451523007s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:17.104459309 +0000 UTC m=+3565.693112967" watchObservedRunningTime="2026-09-09 20:54:17.107581095 +0000 UTC m=+3565.696234755"
Sep 09 20:54:21.080247 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:21.080214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:22.284387 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:22.284331    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:54:22.437229 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:22.437194    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:22.437414 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:22.437255    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:54:22.438807 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:22.438772    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:54:23.949646 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:23.949598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:28.918891 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:28.918851    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:30.816035 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:30.815994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:32.284029 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:32.283973    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:54:32.437083 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:32.437038    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:54:42.284549 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:42.284478    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:54:42.437192 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:42.437145    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:54:42.459618 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:42.459585    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:45.730620 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:45.730528    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:45.830914 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:45.830881    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:52.135208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:52.135176    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:54:52.148431 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:52.148406    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:54:52.283973 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:52.283920    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:54:52.437271 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:54:52.437164    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:55:02.284259 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:02.284210    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:55:02.436966 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:02.436917    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:55:12.284540 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:12.284483    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:55:12.437273 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:12.437226    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:55:22.284454 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:22.284401    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:55:22.437645 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:22.437597    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:55:32.284773 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:32.284719    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:55:32.436948 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:32.436901    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:55:42.284458 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:42.284404    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:55:42.436905 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:42.436857    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:55:52.294248 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:52.294210    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:55:52.302452 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:52.302421    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:55:52.437297 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:52.437254    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:55:53.851262 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:53.851220    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"]
Sep 09 20:55:53.863599 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:53.863557    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:53.940146 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:55:53.940112    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:02.436825 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:02.436775    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:56:05.768341 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:05.768294    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:08.874326 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:08.874274    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" containerID="cri-o://952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1" gracePeriod=45
Sep 09 20:56:09.294946 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.294832    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 09 20:56:09.833142 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.833110    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:56:09.903343 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.903300    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tmp-dir\") pod \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") "
Sep 09 20:56:09.903808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.903356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-dshm\") pod \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") "
Sep 09 20:56:09.903808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.903379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kserve-provision-location\") pod \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") "
Sep 09 20:56:09.903808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.903401    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-model-cache\") pod \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") "
Sep 09 20:56:09.903808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.903427    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-cabundle-cert\") pod \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") "
Sep 09 20:56:09.903808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.903453    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tls-certs\") pod \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") "
Sep 09 20:56:09.903808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.903511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bblv\" (UniqueName: \"kubernetes.io/projected/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kube-api-access-5bblv\") pod \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") "
Sep 09 20:56:09.903808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.903535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-home\") pod \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\" (UID: \"7b9dd516-d1fd-4e99-99a1-0238e38dacc9\") "
Sep 09 20:56:09.903808 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.903697    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b9dd516-d1fd-4e99-99a1-0238e38dacc9" (UID: "7b9dd516-d1fd-4e99-99a1-0238e38dacc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:09.904257 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.904100    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7b9dd516-d1fd-4e99-99a1-0238e38dacc9" (UID: "7b9dd516-d1fd-4e99-99a1-0238e38dacc9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:09.904257 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.904119    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:09.904505 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.904460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-home" (OuterVolumeSpecName: "home") pod "7b9dd516-d1fd-4e99-99a1-0238e38dacc9" (UID: "7b9dd516-d1fd-4e99-99a1-0238e38dacc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:09.906553 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.906525    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-dshm" (OuterVolumeSpecName: "dshm") pod "7b9dd516-d1fd-4e99-99a1-0238e38dacc9" (UID: "7b9dd516-d1fd-4e99-99a1-0238e38dacc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:09.906673 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.906646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kube-api-access-5bblv" (OuterVolumeSpecName: "kube-api-access-5bblv") pod "7b9dd516-d1fd-4e99-99a1-0238e38dacc9" (UID: "7b9dd516-d1fd-4e99-99a1-0238e38dacc9"). InnerVolumeSpecName "kube-api-access-5bblv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:09.906942 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.906919    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b9dd516-d1fd-4e99-99a1-0238e38dacc9" (UID: "7b9dd516-d1fd-4e99-99a1-0238e38dacc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:09.915427 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.915397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b9dd516-d1fd-4e99-99a1-0238e38dacc9" (UID: "7b9dd516-d1fd-4e99-99a1-0238e38dacc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:09.933547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:09.933499    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b9dd516-d1fd-4e99-99a1-0238e38dacc9" (UID: "7b9dd516-d1fd-4e99-99a1-0238e38dacc9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:10.004764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.004728    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:10.004764 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.004766    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:10.004930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.004779    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bblv\" (UniqueName: \"kubernetes.io/projected/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kube-api-access-5bblv\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:10.004930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.004792    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:10.004930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.004803    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:10.004930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.004816    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:10.004930 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.004827    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b9dd516-d1fd-4e99-99a1-0238e38dacc9-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:10.665122 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.665080    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerID="952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1" exitCode=0
Sep 09 20:56:10.665332 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.665130    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"
Sep 09 20:56:10.665332 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.665149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" event={"ID":"7b9dd516-d1fd-4e99-99a1-0238e38dacc9","Type":"ContainerDied","Data":"952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1"}
Sep 09 20:56:10.665332 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.665208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw" event={"ID":"7b9dd516-d1fd-4e99-99a1-0238e38dacc9","Type":"ContainerDied","Data":"171a0bfdb1de06251be0757db66817dc63024757275e19fd635625d9f3c620e4"}
Sep 09 20:56:10.665332 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.665236    2578 scope.go:117] "RemoveContainer" containerID="952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1"
Sep 09 20:56:10.675206 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.675186    2578 scope.go:117] "RemoveContainer" containerID="0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec"
Sep 09 20:56:10.683462 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.683442    2578 scope.go:117] "RemoveContainer" containerID="952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1"
Sep 09 20:56:10.683744 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:56:10.683724    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1\": container with ID starting with 952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1 not found: ID does not exist" containerID="952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1"
Sep 09 20:56:10.683784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.683752    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1"} err="failed to get container status \"952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1\": rpc error: code = NotFound desc = could not find container \"952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1\": container with ID starting with 952f814af4a6b09df53c0a9101af1fd568049e8250f3ffc233e5b41e2d617df1 not found: ID does not exist"
Sep 09 20:56:10.683784 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.683768    2578 scope.go:117] "RemoveContainer" containerID="0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec"
Sep 09 20:56:10.684010 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:56:10.683989    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec\": container with ID starting with 0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec not found: ID does not exist" containerID="0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec"
Sep 09 20:56:10.684074 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.684040    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec"} err="failed to get container status \"0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec\": rpc error: code = NotFound desc = could not find container \"0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec\": container with ID starting with 0eac7298fd55be817904c693e1f861531ebc083d5393f4491fc2dfa46a3060ec not found: ID does not exist"
Sep 09 20:56:10.689210 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.689185    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"]
Sep 09 20:56:10.693228 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:10.693204    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-cd46bf7bc-nx9mw"]
Sep 09 20:56:11.895533 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:11.895496    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" path="/var/lib/kubelet/pods/7b9dd516-d1fd-4e99-99a1-0238e38dacc9/volumes"
Sep 09 20:56:12.403361 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:12.403320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:12.436625 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:12.436584    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 09 20:56:12.505647 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:12.505607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:22.447007 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:22.446910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:56:22.455379 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:22.455333    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:56:23.438895 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:23.438860    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:32.399519 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:32.399481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:32.416067 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:32.416037    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"]
Sep 09 20:56:32.471724 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:32.471691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:33.530374 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:33.530322    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:33.592894 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:33.592859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:43.920116 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:43.920080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:44.024548 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:44.024496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:47.439434 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:47.439369    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main" containerID="cri-o://c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660" gracePeriod=45
Sep 09 20:56:48.185743 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.185714    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:56:48.227908 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.227806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-cabundle-cert\") pod \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") "
Sep 09 20:56:48.227908 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.227848    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kserve-provision-location\") pod \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") "
Sep 09 20:56:48.227908 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.227868    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-model-cache\") pod \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") "
Sep 09 20:56:48.228208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.227919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tmp-dir\") pod \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") "
Sep 09 20:56:48.228208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.227938    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-home\") pod \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") "
Sep 09 20:56:48.228208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.227963    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tls-certs\") pod \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") "
Sep 09 20:56:48.228208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.227992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gg2c\" (UniqueName: \"kubernetes.io/projected/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kube-api-access-5gg2c\") pod \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") "
Sep 09 20:56:48.228208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.228075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-dshm\") pod \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\" (UID: \"d453f43a-682f-444a-bd4a-0cba8b3ea8b2\") "
Sep 09 20:56:48.228208 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.228195    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "d453f43a-682f-444a-bd4a-0cba8b3ea8b2" (UID: "d453f43a-682f-444a-bd4a-0cba8b3ea8b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:48.228965 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.228333    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:48.228965 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.228339    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d453f43a-682f-444a-bd4a-0cba8b3ea8b2" (UID: "d453f43a-682f-444a-bd4a-0cba8b3ea8b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:48.228965 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.228573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-home" (OuterVolumeSpecName: "home") pod "d453f43a-682f-444a-bd4a-0cba8b3ea8b2" (UID: "d453f43a-682f-444a-bd4a-0cba8b3ea8b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:48.230575 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.230536    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kube-api-access-5gg2c" (OuterVolumeSpecName: "kube-api-access-5gg2c") pod "d453f43a-682f-444a-bd4a-0cba8b3ea8b2" (UID: "d453f43a-682f-444a-bd4a-0cba8b3ea8b2"). InnerVolumeSpecName "kube-api-access-5gg2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:48.230942 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.230909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d453f43a-682f-444a-bd4a-0cba8b3ea8b2" (UID: "d453f43a-682f-444a-bd4a-0cba8b3ea8b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:48.231020 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.230939    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-dshm" (OuterVolumeSpecName: "dshm") pod "d453f43a-682f-444a-bd4a-0cba8b3ea8b2" (UID: "d453f43a-682f-444a-bd4a-0cba8b3ea8b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:48.247617 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.247583    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d453f43a-682f-444a-bd4a-0cba8b3ea8b2" (UID: "d453f43a-682f-444a-bd4a-0cba8b3ea8b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:48.256893 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.256867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d453f43a-682f-444a-bd4a-0cba8b3ea8b2" (UID: "d453f43a-682f-444a-bd4a-0cba8b3ea8b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:48.329629 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.329593    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:48.329629 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.329625    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:48.329820 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.329638    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:48.329820 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.329647    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:48.329820 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.329655    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:48.329820 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.329662    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:48.329820 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.329669    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gg2c\" (UniqueName: \"kubernetes.io/projected/d453f43a-682f-444a-bd4a-0cba8b3ea8b2-kube-api-access-5gg2c\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 20:56:48.832408 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.832367    2578 generic.go:356] "Generic (PLEG): container finished" podID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerID="c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660" exitCode=0
Sep 09 20:56:48.832807 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.832423    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"
Sep 09 20:56:48.832807 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.832452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" event={"ID":"d453f43a-682f-444a-bd4a-0cba8b3ea8b2","Type":"ContainerDied","Data":"c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660"}
Sep 09 20:56:48.832807 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.832511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf" event={"ID":"d453f43a-682f-444a-bd4a-0cba8b3ea8b2","Type":"ContainerDied","Data":"c2a4bb88c1844b74570ae23f847a8b47475fba42eb97b8493e3c65b1e31472c3"}
Sep 09 20:56:48.832807 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.832533    2578 scope.go:117] "RemoveContainer" containerID="c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660"
Sep 09 20:56:48.842070 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.842051    2578 scope.go:117] "RemoveContainer" containerID="a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369"
Sep 09 20:56:48.851879 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.851855    2578 scope.go:117] "RemoveContainer" containerID="c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660"
Sep 09 20:56:48.852415 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:56:48.852391    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660\": container with ID starting with c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660 not found: ID does not exist" containerID="c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660"
Sep 09 20:56:48.852525 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.852426    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660"} err="failed to get container status \"c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660\": rpc error: code = NotFound desc = could not find container \"c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660\": container with ID starting with c22f620352c5620b9175f1ad14a75e288c9c6a18863158c1fb81bfa686959660 not found: ID does not exist"
Sep 09 20:56:48.852525 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.852450    2578 scope.go:117] "RemoveContainer" containerID="a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369"
Sep 09 20:56:48.852750 ip-10-0-138-141 kubenswrapper[2578]: E0909 20:56:48.852730    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369\": container with ID starting with a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369 not found: ID does not exist" containerID="a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369"
Sep 09 20:56:48.852794 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.852756    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369"} err="failed to get container status \"a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369\": rpc error: code = NotFound desc = could not find container \"a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369\": container with ID starting with a5d0051bb32e9a55c228448c961635abeddb3de8c034bbb431f39b3dc6a36369 not found: ID does not exist"
Sep 09 20:56:48.859944 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.859924    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"]
Sep 09 20:56:48.864600 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:48.864578    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-74d68kf"]
Sep 09 20:56:49.895823 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:49.895790    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" path="/var/lib/kubelet/pods/d453f43a-682f-444a-bd4a-0cba8b3ea8b2/volumes"
Sep 09 20:56:50.920018 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:50.919987    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:56.552223 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:56.552175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:58.526096 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:58.526066    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:58.634187 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:56:58.634143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:04.670431 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:04.670390    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:22.581495 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:22.581457    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:22.676041 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:22.676012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:29.264329 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:29.264269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:34.939760 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:34.939723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:35.039899 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:35.039865    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:43.296767 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.296733    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"]
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297033    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="storage-initializer"
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297044    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="storage-initializer"
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297056    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="storage-initializer"
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297061    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="storage-initializer"
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297072    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main"
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297078    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main"
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297085    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main"
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297090    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main"
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297157    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b9dd516-d1fd-4e99-99a1-0238e38dacc9" containerName="main"
Sep 09 20:57:43.297233 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.297165    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d453f43a-682f-444a-bd4a-0cba8b3ea8b2" containerName="main"
Sep 09 20:57:43.300201 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.300183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.303457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.303422    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:57:43.303597 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.303494    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 09 20:57:43.303735 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.303717    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 09 20:57:43.303953 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.303928    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cwr8d\""
Sep 09 20:57:43.304069 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.303928    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 09 20:57:43.304394 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.304376    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-zslrw\""
Sep 09 20:57:43.304495 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.304398    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 09 20:57:43.317064 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.317031    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"]
Sep 09 20:57:43.349512 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.349474    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:43.395547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.395510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-model-cache\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.395547 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.395545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6992f72-9bca-48bd-8432-4d4c57b86b98-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.395818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.395566    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-dshm\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.395818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.395672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.395818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.395722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6992f72-9bca-48bd-8432-4d4c57b86b98-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.395818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.395749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-home\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.395818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.395783    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.395818 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.395809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26s8w\" (UniqueName: \"kubernetes.io/projected/c6992f72-9bca-48bd-8432-4d4c57b86b98-kube-api-access-26s8w\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496194 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6992f72-9bca-48bd-8432-4d4c57b86b98-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496194 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-home\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26s8w\" (UniqueName: \"kubernetes.io/projected/c6992f72-9bca-48bd-8432-4d4c57b86b98-kube-api-access-26s8w\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-model-cache\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496355    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6992f72-9bca-48bd-8432-4d4c57b86b98-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-dshm\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496457 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496790 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-home\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496790 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.496790 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-model-cache\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.497016 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.496993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.497083 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.497036    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6992f72-9bca-48bd-8432-4d4c57b86b98-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.498696 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.498658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-dshm\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.498962 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.498943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6992f72-9bca-48bd-8432-4d4c57b86b98-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.515192 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.515171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26s8w\" (UniqueName: \"kubernetes.io/projected/c6992f72-9bca-48bd-8432-4d4c57b86b98-kube-api-access-26s8w\") pod \"custom-route-timeout-pd-test-kserve-55849689c6-n6zgm\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.611697 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.611656    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:43.631252 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.631217    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:43.894739 ip-10-0-138-141 kubenswrapper[2578]: W0909 20:57:43.894705    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6992f72_9bca_48bd_8432_4d4c57b86b98.slice/crio-1d9bfd29f13a9c4b64edc568770fdd6ffbb8670ea77eebcdbadfdef9ce0793d8 WatchSource:0}: Error finding container 1d9bfd29f13a9c4b64edc568770fdd6ffbb8670ea77eebcdbadfdef9ce0793d8: Status 404 returned error can't find the container with id 1d9bfd29f13a9c4b64edc568770fdd6ffbb8670ea77eebcdbadfdef9ce0793d8
Sep 09 20:57:43.897083 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:43.896999    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"]
Sep 09 20:57:44.080440 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:44.080343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" event={"ID":"c6992f72-9bca-48bd-8432-4d4c57b86b98","Type":"ContainerStarted","Data":"ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93"}
Sep 09 20:57:44.080440 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:44.080419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" event={"ID":"c6992f72-9bca-48bd-8432-4d4c57b86b98","Type":"ContainerStarted","Data":"1d9bfd29f13a9c4b64edc568770fdd6ffbb8670ea77eebcdbadfdef9ce0793d8"}
Sep 09 20:57:44.080673 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:44.080535    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:45.086713 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:45.086665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" event={"ID":"c6992f72-9bca-48bd-8432-4d4c57b86b98","Type":"ContainerStarted","Data":"1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708"}
Sep 09 20:57:47.098719 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:47.098674    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerID="1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708" exitCode=0
Sep 09 20:57:47.099146 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:47.098745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" event={"ID":"c6992f72-9bca-48bd-8432-4d4c57b86b98","Type":"ContainerDied","Data":"1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708"}
Sep 09 20:57:48.106595 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:48.106556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" event={"ID":"c6992f72-9bca-48bd-8432-4d4c57b86b98","Type":"ContainerStarted","Data":"b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6"}
Sep 09 20:57:48.134969 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:48.134910    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podStartSLOduration=2.90593219 podStartE2EDuration="5.134893279s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.228961089s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:48.133218269 +0000 UTC m=+3776.721871937" watchObservedRunningTime="2026-09-09 20:57:48.134893279 +0000 UTC m=+3776.723546933"
Sep 09 20:57:53.612165 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:53.612110    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:53.612823 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:53.612192    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:53.613559 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:53.613526    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:57:53.625888 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:53.625861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:57:56.079766 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:57:56.079723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:03.612761 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:58:03.612689    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:58:13.612354 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:58:13.612274    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:58:23.612446 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:58:23.612385    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:58:33.613209 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:58:33.613156    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:58:43.612876 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:58:43.612830    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:58:53.612934 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:58:53.612882    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:58:58.359640 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:58:58.359603    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:03.612548 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:03.612489    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:59:13.613038 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:13.612964    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:59:18.309896 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:18.309861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:18.437136 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:18.437093    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:18.574912 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:18.574827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:23.612393 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:23.612345    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:59:24.566235 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:24.566193    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:24.629333 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:24.629288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:33.612350 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:33.612284    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8001/health\": dial tcp 10.132.0.77:8001: connect: connection refused"
Sep 09 20:59:37.804273 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:37.804230    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:43.622143 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:43.622108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:59:43.638141 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:43.638115    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 20:59:52.169758 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:52.169723    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 20:59:52.187285 ip-10-0-138-141 kubenswrapper[2578]: I0909 20:59:52.187256    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 21:00:03.210130 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:03.210093    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"]
Sep 09 21:00:03.703580 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:03.703541    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:03.823968 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:03.823933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:18.233553 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:18.233511    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main" containerID="cri-o://b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6" gracePeriod=45
Sep 09 21:00:18.850969 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:18.850912    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="llm-d-routing-sidecar" containerID="cri-o://ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93" gracePeriod=45
Sep 09 21:00:19.098084 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.098055    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 21:00:19.151753 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.151717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-model-cache\") pod \"c6992f72-9bca-48bd-8432-4d4c57b86b98\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") "
Sep 09 21:00:19.151968 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.151762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-kserve-provision-location\") pod \"c6992f72-9bca-48bd-8432-4d4c57b86b98\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") "
Sep 09 21:00:19.151968 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.151803    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6992f72-9bca-48bd-8432-4d4c57b86b98-tls-certs\") pod \"c6992f72-9bca-48bd-8432-4d4c57b86b98\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") "
Sep 09 21:00:19.151968 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.151865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-dshm\") pod \"c6992f72-9bca-48bd-8432-4d4c57b86b98\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") "
Sep 09 21:00:19.151968 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.151914    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26s8w\" (UniqueName: \"kubernetes.io/projected/c6992f72-9bca-48bd-8432-4d4c57b86b98-kube-api-access-26s8w\") pod \"c6992f72-9bca-48bd-8432-4d4c57b86b98\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") "
Sep 09 21:00:19.151968 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.151951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-tmp-dir\") pod \"c6992f72-9bca-48bd-8432-4d4c57b86b98\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") "
Sep 09 21:00:19.152263 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.151998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6992f72-9bca-48bd-8432-4d4c57b86b98-cabundle-cert\") pod \"c6992f72-9bca-48bd-8432-4d4c57b86b98\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") "
Sep 09 21:00:19.152263 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.152008    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6992f72-9bca-48bd-8432-4d4c57b86b98" (UID: "c6992f72-9bca-48bd-8432-4d4c57b86b98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:19.152263 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.152022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-home\") pod \"c6992f72-9bca-48bd-8432-4d4c57b86b98\" (UID: \"c6992f72-9bca-48bd-8432-4d4c57b86b98\") "
Sep 09 21:00:19.152444 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.152362    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:00:19.152727 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.152700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6992f72-9bca-48bd-8432-4d4c57b86b98-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6992f72-9bca-48bd-8432-4d4c57b86b98" (UID: "c6992f72-9bca-48bd-8432-4d4c57b86b98"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:19.152830 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.152768    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-home" (OuterVolumeSpecName: "home") pod "c6992f72-9bca-48bd-8432-4d4c57b86b98" (UID: "c6992f72-9bca-48bd-8432-4d4c57b86b98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:19.154543 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.154504    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-dshm" (OuterVolumeSpecName: "dshm") pod "c6992f72-9bca-48bd-8432-4d4c57b86b98" (UID: "c6992f72-9bca-48bd-8432-4d4c57b86b98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:19.154674 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.154532    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6992f72-9bca-48bd-8432-4d4c57b86b98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6992f72-9bca-48bd-8432-4d4c57b86b98" (UID: "c6992f72-9bca-48bd-8432-4d4c57b86b98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:19.154867 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.154847    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6992f72-9bca-48bd-8432-4d4c57b86b98-kube-api-access-26s8w" (OuterVolumeSpecName: "kube-api-access-26s8w") pod "c6992f72-9bca-48bd-8432-4d4c57b86b98" (UID: "c6992f72-9bca-48bd-8432-4d4c57b86b98"). InnerVolumeSpecName "kube-api-access-26s8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:19.166253 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.166218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6992f72-9bca-48bd-8432-4d4c57b86b98" (UID: "c6992f72-9bca-48bd-8432-4d4c57b86b98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:19.179471 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.179436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6992f72-9bca-48bd-8432-4d4c57b86b98" (UID: "c6992f72-9bca-48bd-8432-4d4c57b86b98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:19.253543 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.253503    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:00:19.253543 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.253533    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6992f72-9bca-48bd-8432-4d4c57b86b98-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:00:19.253543 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.253542    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:00:19.253543 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.253553    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:00:19.254038 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.253562    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6992f72-9bca-48bd-8432-4d4c57b86b98-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:00:19.254038 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.253570    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6992f72-9bca-48bd-8432-4d4c57b86b98-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:00:19.254038 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.253578    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26s8w\" (UniqueName: \"kubernetes.io/projected/c6992f72-9bca-48bd-8432-4d4c57b86b98-kube-api-access-26s8w\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:00:19.821979 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.821935    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerID="b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6" exitCode=0
Sep 09 21:00:19.821979 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.821967    2578 generic.go:356] "Generic (PLEG): container finished" podID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerID="ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93" exitCode=0
Sep 09 21:00:19.822252 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.822002    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"
Sep 09 21:00:19.822252 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.822011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" event={"ID":"c6992f72-9bca-48bd-8432-4d4c57b86b98","Type":"ContainerDied","Data":"b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6"}
Sep 09 21:00:19.822252 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.822065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" event={"ID":"c6992f72-9bca-48bd-8432-4d4c57b86b98","Type":"ContainerDied","Data":"ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93"}
Sep 09 21:00:19.822252 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.822085    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm" event={"ID":"c6992f72-9bca-48bd-8432-4d4c57b86b98","Type":"ContainerDied","Data":"1d9bfd29f13a9c4b64edc568770fdd6ffbb8670ea77eebcdbadfdef9ce0793d8"}
Sep 09 21:00:19.822252 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.822106    2578 scope.go:117] "RemoveContainer" containerID="b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6"
Sep 09 21:00:19.831657 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.831635    2578 scope.go:117] "RemoveContainer" containerID="1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708"
Sep 09 21:00:19.839827 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.839807    2578 scope.go:117] "RemoveContainer" containerID="ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93"
Sep 09 21:00:19.846857 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.846829    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"]
Sep 09 21:00:19.848287 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.848272    2578 scope.go:117] "RemoveContainer" containerID="b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6"
Sep 09 21:00:19.848610 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:00:19.848589    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6\": container with ID starting with b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6 not found: ID does not exist" containerID="b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6"
Sep 09 21:00:19.848714 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.848624    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6"} err="failed to get container status \"b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6\": rpc error: code = NotFound desc = could not find container \"b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6\": container with ID starting with b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6 not found: ID does not exist"
Sep 09 21:00:19.848714 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.848650    2578 scope.go:117] "RemoveContainer" containerID="1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708"
Sep 09 21:00:19.848903 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:00:19.848884    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708\": container with ID starting with 1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708 not found: ID does not exist" containerID="1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708"
Sep 09 21:00:19.849001 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.848911    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708"} err="failed to get container status \"1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708\": rpc error: code = NotFound desc = could not find container \"1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708\": container with ID starting with 1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708 not found: ID does not exist"
Sep 09 21:00:19.849001 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.848934    2578 scope.go:117] "RemoveContainer" containerID="ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93"
Sep 09 21:00:19.849270 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:00:19.849237    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93\": container with ID starting with ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93 not found: ID does not exist" containerID="ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93"
Sep 09 21:00:19.849345 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.849280    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93"} err="failed to get container status \"ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93\": rpc error: code = NotFound desc = could not find container \"ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93\": container with ID starting with ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93 not found: ID does not exist"
Sep 09 21:00:19.849345 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.849301    2578 scope.go:117] "RemoveContainer" containerID="b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6"
Sep 09 21:00:19.849642 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.849612    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6"} err="failed to get container status \"b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6\": rpc error: code = NotFound desc = could not find container \"b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6\": container with ID starting with b8eb86ec1468ec567b13dd4c36dfd5f3126820d3ab7a9ba4e5f920f73d1a69e6 not found: ID does not exist"
Sep 09 21:00:19.849642 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.849642    2578 scope.go:117] "RemoveContainer" containerID="1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708"
Sep 09 21:00:19.849948 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.849911    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708"} err="failed to get container status \"1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708\": rpc error: code = NotFound desc = could not find container \"1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708\": container with ID starting with 1d0887ca6ec9a607743f10636a5b0813b5a408d36f1c5731dc3a8e4b9a57b708 not found: ID does not exist"
Sep 09 21:00:19.849948 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.849934    2578 scope.go:117] "RemoveContainer" containerID="ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93"
Sep 09 21:00:19.850211 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.850192    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93"} err="failed to get container status \"ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93\": rpc error: code = NotFound desc = could not find container \"ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93\": container with ID starting with ea7dc83b4d952f53144473fd61f760453e3a523360f64d1ae8008023a9ecbc93 not found: ID does not exist"
Sep 09 21:00:19.851752 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.851732    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-55849689c6-n6zgm"]
Sep 09 21:00:19.896670 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:19.896640    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" path="/var/lib/kubelet/pods/c6992f72-9bca-48bd-8432-4d4c57b86b98/volumes"
Sep 09 21:00:53.376829 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.376793    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:53.527754 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.527716    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:53.836614 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.836576    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"]
Sep 09 21:00:53.836903 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.836889    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main"
Sep 09 21:00:53.836966 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.836904    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main"
Sep 09 21:00:53.836966 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.836917    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="storage-initializer"
Sep 09 21:00:53.836966 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.836923    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="storage-initializer"
Sep 09 21:00:53.836966 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.836937    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="llm-d-routing-sidecar"
Sep 09 21:00:53.836966 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.836943    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="llm-d-routing-sidecar"
Sep 09 21:00:53.837146 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.836995    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="llm-d-routing-sidecar"
Sep 09 21:00:53.837146 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.837007    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6992f72-9bca-48bd-8432-4d4c57b86b98" containerName="main"
Sep 09 21:00:53.838790 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.838772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:53.841900 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.841877    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 09 21:00:53.842920 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.842896    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-hkd8v\""
Sep 09 21:00:53.863561 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.863535    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 09 21:00:53.873319 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.873280    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 09 21:00:53.875022 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.875001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"]
Sep 09 21:00:53.914931 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.914899    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:53.946283 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.946246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:53.946283 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.946283    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb073d3-8a01-409c-adb3-8163bedb0db3-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:53.946511 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.946326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97954\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-kube-api-access-97954\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:53.946511 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.946385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:53.946511 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.946422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:53.946511 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.946471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:53.946511 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.946504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:53.946687 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.946524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:53.946687 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:53.946552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.027017 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.026982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:54.047211 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047176    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047424 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb073d3-8a01-409c-adb3-8163bedb0db3-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047424 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97954\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-kube-api-access-97954\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047424 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047283    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047424 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047424 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047424 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047760 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047431    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047760 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047760 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047922 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047818    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.047922 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.047854    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.048049 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.048023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.048113 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.048058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb073d3-8a01-409c-adb3-8163bedb0db3-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.049854 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.049835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.050074 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.050056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.058970 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.058918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97954\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-kube-api-access-97954\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.061063 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.061044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-cpfwb\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.148432 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.148390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:54.425873 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.425788    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"]
Sep 09 21:00:54.429403 ip-10-0-138-141 kubenswrapper[2578]: W0909 21:00:54.429374    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb073d3_8a01_409c_adb3_8163bedb0db3.slice/crio-cd6ad05dc51be1fc87fbfde01aa9861d666236830bc030b91e9d9ec406231987 WatchSource:0}: Error finding container cd6ad05dc51be1fc87fbfde01aa9861d666236830bc030b91e9d9ec406231987: Status 404 returned error can't find the container with id cd6ad05dc51be1fc87fbfde01aa9861d666236830bc030b91e9d9ec406231987
Sep 09 21:00:54.431435 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.431398    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:00:54.431522 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.431463    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:00:54.431522 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.431492    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:00:54.985116 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.985077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" event={"ID":"6fb073d3-8a01-409c-adb3-8163bedb0db3","Type":"ContainerStarted","Data":"9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b"}
Sep 09 21:00:54.985302 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:54.985130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" event={"ID":"6fb073d3-8a01-409c-adb3-8163bedb0db3","Type":"ContainerStarted","Data":"cd6ad05dc51be1fc87fbfde01aa9861d666236830bc030b91e9d9ec406231987"}
Sep 09 21:00:55.011098 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:55.011042    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" podStartSLOduration=2.011025701 podStartE2EDuration="2.011025701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:55.008703304 +0000 UTC m=+3963.597356961" watchObservedRunningTime="2026-09-09 21:00:55.011025701 +0000 UTC m=+3963.599679358"
Sep 09 21:00:55.149199 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:55.149160    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:55.154086 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:55.154056    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:55.990850 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:55.990809    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:55.992063 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:55.992038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:00:58.090762 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:00:58.090719    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:00.362061 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:00.362019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:05.896475 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:05.896441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:05.965659 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:05.965630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:06.043806 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.043777    2578 generic.go:356] "Generic (PLEG): container finished" podID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerID="9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b" exitCode=0
Sep 09 21:01:06.043944 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.043848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" event={"ID":"6fb073d3-8a01-409c-adb3-8163bedb0db3","Type":"ContainerDied","Data":"9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b"}
Sep 09 21:01:06.044226 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.044208    2578 scope.go:117] "RemoveContainer" containerID="9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b"
Sep 09 21:01:06.045218 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.045189    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:01:06.045294 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.045276    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:01:06.045395 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.045377    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 09 21:01:06.889344 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.889287    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"]
Sep 09 21:01:06.891741 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.891721    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:06.895734 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.895711    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-thxvh\""
Sep 09 21:01:06.895983 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.895966    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-qthx4\""
Sep 09 21:01:06.896080 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.896060    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 09 21:01:06.896144 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.896131    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 09 21:01:06.896641 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.896626    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:01:06.914570 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.914541    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"]
Sep 09 21:01:06.941359 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.941292    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"]
Sep 09 21:01:06.944143 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.944117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:06.957903 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.957858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:06.958116 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.957916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-dshm\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:06.958116 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.957961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-model-cache\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:06.958116 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.958022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tmp-dir\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:06.958116 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.958052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:06.958116 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.958100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tls-certs\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:06.958432 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.958144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmrvw\" (UniqueName: \"kubernetes.io/projected/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kube-api-access-fmrvw\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:06.958432 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.958190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-home\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:06.961617 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.961587    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"]
Sep 09 21:01:06.977199 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:06.977154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:07.066328 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.066165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" event={"ID":"6fb073d3-8a01-409c-adb3-8163bedb0db3","Type":"ContainerStarted","Data":"06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e"}
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.067804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.067846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41d5f3e6-67f7-432f-825a-284ee20acb77-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.067883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.067911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-dshm\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.067935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-model-cache\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.067963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d5f3e6-67f7-432f-825a-284ee20acb77-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.067995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.068031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tmp-dir\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.068060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.068086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.068121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tls-certs\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.068331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.068148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlqc\" (UniqueName: \"kubernetes.io/projected/41d5f3e6-67f7-432f-825a-284ee20acb77-kube-api-access-ltlqc\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.069071 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.068361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmrvw\" (UniqueName: \"kubernetes.io/projected/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kube-api-access-fmrvw\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.069071 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.068417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-home\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.069071 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.068448    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-home\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.069071 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.068482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.069268 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.069188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-home\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.069685 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.069660    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.069873 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.069731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-model-cache\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.070286 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.070259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.070427 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.070282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tmp-dir\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.072029 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.072004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-dshm\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.073434 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.073415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tls-certs\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.077348 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.077323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmrvw\" (UniqueName: \"kubernetes.io/projected/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kube-api-access-fmrvw\") pod \"router-with-refs-pd-test-kserve-556656f58d-4svlx\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.149394 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.149270    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:01:07.150987 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.150695    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 09 21:01:07.169171 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d5f3e6-67f7-432f-825a-284ee20acb77-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.169409 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.169409 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.169409 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlqc\" (UniqueName: \"kubernetes.io/projected/41d5f3e6-67f7-432f-825a-284ee20acb77-kube-api-access-ltlqc\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.169409 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-home\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.169409 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.169709 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.169709 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41d5f3e6-67f7-432f-825a-284ee20acb77-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.169823 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.169960 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-home\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.170031 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.169993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.170125 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.170102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.170200 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.170178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41d5f3e6-67f7-432f-825a-284ee20acb77-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.172056 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.172028    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d5f3e6-67f7-432f-825a-284ee20acb77-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.172361 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.172340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.178011 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.177981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlqc\" (UniqueName: \"kubernetes.io/projected/41d5f3e6-67f7-432f-825a-284ee20acb77-kube-api-access-ltlqc\") pod \"router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.201883 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.201852    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:07.277042 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.276768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:07.432754 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.432668    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:07.503867 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.503839    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"]
Sep 09 21:01:07.505793 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.505766    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:07.506095 ip-10-0-138-141 kubenswrapper[2578]: W0909 21:01:07.506072    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e617d1c_9103_4aa6_a43e_aaa84c55ae57.slice/crio-f3b8795f180b4679e976a71d9f24449c355778c8a7aa2309fffbb4a31174c0b4 WatchSource:0}: Error finding container f3b8795f180b4679e976a71d9f24449c355778c8a7aa2309fffbb4a31174c0b4: Status 404 returned error can't find the container with id f3b8795f180b4679e976a71d9f24449c355778c8a7aa2309fffbb4a31174c0b4
Sep 09 21:01:07.559565 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:07.559430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"]
Sep 09 21:01:07.565585 ip-10-0-138-141 kubenswrapper[2578]: W0909 21:01:07.565550    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d5f3e6_67f7_432f_825a_284ee20acb77.slice/crio-c3faed9a3fc5ffea089882ed32cf07e64be47a50c453b79b7243fabf57fcb945 WatchSource:0}: Error finding container c3faed9a3fc5ffea089882ed32cf07e64be47a50c453b79b7243fabf57fcb945: Status 404 returned error can't find the container with id c3faed9a3fc5ffea089882ed32cf07e64be47a50c453b79b7243fabf57fcb945
Sep 09 21:01:08.074007 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:08.073961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" event={"ID":"2e617d1c-9103-4aa6-a43e-aaa84c55ae57","Type":"ContainerStarted","Data":"b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976"}
Sep 09 21:01:08.074539 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:08.074038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" event={"ID":"2e617d1c-9103-4aa6-a43e-aaa84c55ae57","Type":"ContainerStarted","Data":"f3b8795f180b4679e976a71d9f24449c355778c8a7aa2309fffbb4a31174c0b4"}
Sep 09 21:01:08.074539 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:08.074126    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:08.075932 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:08.075907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" event={"ID":"41d5f3e6-67f7-432f-825a-284ee20acb77","Type":"ContainerStarted","Data":"8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29"}
Sep 09 21:01:08.076048 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:08.075948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" event={"ID":"41d5f3e6-67f7-432f-825a-284ee20acb77","Type":"ContainerStarted","Data":"c3faed9a3fc5ffea089882ed32cf07e64be47a50c453b79b7243fabf57fcb945"}
Sep 09 21:01:08.149069 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:08.149024    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 09 21:01:09.088886 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:09.088847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" event={"ID":"2e617d1c-9103-4aa6-a43e-aaa84c55ae57","Type":"ContainerStarted","Data":"2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7"}
Sep 09 21:01:09.149853 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:09.149798    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 09 21:01:10.150187 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:10.149720    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 09 21:01:10.991639 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:10.991602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:01:11.099975 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:11.099939    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerID="2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7" exitCode=0
Sep 09 21:01:11.100163 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:11.100014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" event={"ID":"2e617d1c-9103-4aa6-a43e-aaa84c55ae57","Type":"ContainerDied","Data":"2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7"}
Sep 09 21:01:11.101718 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:11.101696    2578 generic.go:356] "Generic (PLEG): container finished" podID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerID="8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29" exitCode=0
Sep 09 21:01:11.101805 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:11.101765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" event={"ID":"41d5f3e6-67f7-432f-825a-284ee20acb77","Type":"ContainerDied","Data":"8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29"}
Sep 09 21:01:11.167497 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:11.164527    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 09 21:01:12.109886 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:12.109844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" event={"ID":"2e617d1c-9103-4aa6-a43e-aaa84c55ae57","Type":"ContainerStarted","Data":"779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3"}
Sep 09 21:01:12.112026 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:12.111998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" event={"ID":"41d5f3e6-67f7-432f-825a-284ee20acb77","Type":"ContainerStarted","Data":"8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7"}
Sep 09 21:01:12.133921 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:12.133865    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podStartSLOduration=3.425490898 podStartE2EDuration="6.133849861s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.708358963s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:12.132865223 +0000 UTC m=+3980.721518879" watchObservedRunningTime="2026-09-09 21:01:12.133849861 +0000 UTC m=+3980.722503517"
Sep 09 21:01:12.149629 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:12.149575    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 09 21:01:12.154080 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:12.154017    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podStartSLOduration=3.451943025 podStartE2EDuration="6.153983801s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.702040776s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:12.153100504 +0000 UTC m=+3980.741754174" watchObservedRunningTime="2026-09-09 21:01:12.153983801 +0000 UTC m=+3980.742637460"
Sep 09 21:01:13.153551 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:13.153518    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:01:13.154859 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:13.154826    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:01:17.202889 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:17.202852    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:17.203371 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:17.202921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:17.204424 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:17.204383    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:01:17.221378 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:17.221346    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:01:17.277405 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:17.277369    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:17.277585 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:17.277433    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:01:17.278916 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:17.278884    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:01:27.202961 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:27.202915    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:01:27.277482 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:27.277426    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:01:37.202842 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:37.202788    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:01:37.277390 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:37.277334    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:01:41.760264 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:41.760215    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:47.202521 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:47.202463    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:01:47.277420 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:47.277367    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:01:57.202696 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:57.202625    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:01:57.277966 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:01:57.277917    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:02:00.444038 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:00.443995    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:00.508416 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:00.508378    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:01.957700 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:01.957662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:02.059695 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:02.059645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:07.024761 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:07.024693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:07.203331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:07.203267    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:02:07.271066 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:07.271030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:07.277740 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:07.277642    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:02:09.377801 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:09.377759    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:09.483740 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:09.483704    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:17.202645 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:17.202537    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:02:17.277324 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:17.277264    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:02:27.202435 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:27.202385    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:02:27.277829 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:27.277773    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:02:37.202720 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:37.202672    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:02:37.277842 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:37.277787    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:02:44.257191 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:44.257155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:47.203283 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:47.203224    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:02:47.277965 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:47.277911    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:02:57.203039 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:57.202986    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:02:57.278199 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:02:57.278148    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:03:07.203158 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:07.203094    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:03:07.278294 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:07.278244    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:03:17.202831 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:17.202772    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:03:17.277436 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:17.277393    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:03:27.203303 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:27.203251    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:03:27.277515 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:27.277458    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:03:36.248232 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:36.248192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:37.202979 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:37.202934    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8001/health\": dial tcp 10.132.0.79:8001: connect: connection refused"
Sep 09 21:03:37.278261 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:37.278209    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 09 21:03:47.212676 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:47.212575    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:03:47.224773 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:47.224746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:03:47.288169 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:47.288132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:03:47.296050 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:47.296014    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:03:56.426059 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:56.426026    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:56.436960 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:56.436932    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"]
Sep 09 21:03:56.442021 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:56.441985    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"]
Sep 09 21:03:56.499865 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:03:56.499835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:11.460418 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:11.460372    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main" containerID="cri-o://8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7" gracePeriod=45
Sep 09 21:04:11.463718 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:11.463683    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main" containerID="cri-o://779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3" gracePeriod=45
Sep 09 21:04:12.280361 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.280323    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="llm-d-routing-sidecar" containerID="cri-o://b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976" gracePeriod=45
Sep 09 21:04:12.414096 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.414071    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:04:12.520198 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.520173    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:04:12.544897 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.544866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltlqc\" (UniqueName: \"kubernetes.io/projected/41d5f3e6-67f7-432f-825a-284ee20acb77-kube-api-access-ltlqc\") pod \"41d5f3e6-67f7-432f-825a-284ee20acb77\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") "
Sep 09 21:04:12.545164 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.545136    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d5f3e6-67f7-432f-825a-284ee20acb77-tls-certs\") pod \"41d5f3e6-67f7-432f-825a-284ee20acb77\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") "
Sep 09 21:04:12.545300 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.545191    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-model-cache\") pod \"41d5f3e6-67f7-432f-825a-284ee20acb77\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") "
Sep 09 21:04:12.545300 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.545224    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-home\") pod \"41d5f3e6-67f7-432f-825a-284ee20acb77\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") "
Sep 09 21:04:12.545485 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.545302    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41d5f3e6-67f7-432f-825a-284ee20acb77-cabundle-cert\") pod \"41d5f3e6-67f7-432f-825a-284ee20acb77\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") "
Sep 09 21:04:12.545485 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.545348    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-tmp-dir\") pod \"41d5f3e6-67f7-432f-825a-284ee20acb77\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") "
Sep 09 21:04:12.545485 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.545380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-kserve-provision-location\") pod \"41d5f3e6-67f7-432f-825a-284ee20acb77\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") "
Sep 09 21:04:12.545485 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.545442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-dshm\") pod \"41d5f3e6-67f7-432f-825a-284ee20acb77\" (UID: \"41d5f3e6-67f7-432f-825a-284ee20acb77\") "
Sep 09 21:04:12.545870 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.545838    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-model-cache" (OuterVolumeSpecName: "model-cache") pod "41d5f3e6-67f7-432f-825a-284ee20acb77" (UID: "41d5f3e6-67f7-432f-825a-284ee20acb77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.546161 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.546127    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d5f3e6-67f7-432f-825a-284ee20acb77-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41d5f3e6-67f7-432f-825a-284ee20acb77" (UID: "41d5f3e6-67f7-432f-825a-284ee20acb77"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:12.546459 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.546401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-home" (OuterVolumeSpecName: "home") pod "41d5f3e6-67f7-432f-825a-284ee20acb77" (UID: "41d5f3e6-67f7-432f-825a-284ee20acb77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.549468 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.549434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d5f3e6-67f7-432f-825a-284ee20acb77-kube-api-access-ltlqc" (OuterVolumeSpecName: "kube-api-access-ltlqc") pod "41d5f3e6-67f7-432f-825a-284ee20acb77" (UID: "41d5f3e6-67f7-432f-825a-284ee20acb77"). InnerVolumeSpecName "kube-api-access-ltlqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:12.549569 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.549526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-dshm" (OuterVolumeSpecName: "dshm") pod "41d5f3e6-67f7-432f-825a-284ee20acb77" (UID: "41d5f3e6-67f7-432f-825a-284ee20acb77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.549784 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.549751    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d5f3e6-67f7-432f-825a-284ee20acb77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41d5f3e6-67f7-432f-825a-284ee20acb77" (UID: "41d5f3e6-67f7-432f-825a-284ee20acb77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:12.564837 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.564792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41d5f3e6-67f7-432f-825a-284ee20acb77" (UID: "41d5f3e6-67f7-432f-825a-284ee20acb77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.575352 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.575325    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41d5f3e6-67f7-432f-825a-284ee20acb77" (UID: "41d5f3e6-67f7-432f-825a-284ee20acb77"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.646512 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.646488    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-dshm\") pod \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") "
Sep 09 21:04:12.646656 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.646518    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmrvw\" (UniqueName: \"kubernetes.io/projected/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kube-api-access-fmrvw\") pod \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") "
Sep 09 21:04:12.646656 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.646539    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-home\") pod \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") "
Sep 09 21:04:12.646656 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.646577    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-model-cache\") pod \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") "
Sep 09 21:04:12.646656 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.646605    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tmp-dir\") pod \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") "
Sep 09 21:04:12.646656 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.646642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tls-certs\") pod \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") "
Sep 09 21:04:12.646952 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.646692    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-cabundle-cert\") pod \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") "
Sep 09 21:04:12.646952 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.646731    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kserve-provision-location\") pod \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\" (UID: \"2e617d1c-9103-4aa6-a43e-aaa84c55ae57\") "
Sep 09 21:04:12.646952 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.646911    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e617d1c-9103-4aa6-a43e-aaa84c55ae57" (UID: "2e617d1c-9103-4aa6-a43e-aaa84c55ae57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.647104 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647069    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.647104 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647088    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41d5f3e6-67f7-432f-825a-284ee20acb77-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.647104 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647102    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.647268 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647115    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.647268 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647128    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.647268 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647132    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e617d1c-9103-4aa6-a43e-aaa84c55ae57" (UID: "2e617d1c-9103-4aa6-a43e-aaa84c55ae57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:12.647268 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647142    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltlqc\" (UniqueName: \"kubernetes.io/projected/41d5f3e6-67f7-432f-825a-284ee20acb77-kube-api-access-ltlqc\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.647268 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647183    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d5f3e6-67f7-432f-825a-284ee20acb77-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.647268 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647200    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.647268 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647212    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d5f3e6-67f7-432f-825a-284ee20acb77-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.647656 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.647299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-home" (OuterVolumeSpecName: "home") pod "2e617d1c-9103-4aa6-a43e-aaa84c55ae57" (UID: "2e617d1c-9103-4aa6-a43e-aaa84c55ae57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.648929 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.648908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-dshm" (OuterVolumeSpecName: "dshm") pod "2e617d1c-9103-4aa6-a43e-aaa84c55ae57" (UID: "2e617d1c-9103-4aa6-a43e-aaa84c55ae57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.649076 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.649057    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e617d1c-9103-4aa6-a43e-aaa84c55ae57" (UID: "2e617d1c-9103-4aa6-a43e-aaa84c55ae57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:12.649148 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.649115    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kube-api-access-fmrvw" (OuterVolumeSpecName: "kube-api-access-fmrvw") pod "2e617d1c-9103-4aa6-a43e-aaa84c55ae57" (UID: "2e617d1c-9103-4aa6-a43e-aaa84c55ae57"). InnerVolumeSpecName "kube-api-access-fmrvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:12.660194 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.660153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e617d1c-9103-4aa6-a43e-aaa84c55ae57" (UID: "2e617d1c-9103-4aa6-a43e-aaa84c55ae57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.671387 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.671358    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e617d1c-9103-4aa6-a43e-aaa84c55ae57" (UID: "2e617d1c-9103-4aa6-a43e-aaa84c55ae57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:12.748087 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.748011    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.748087 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.748036    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.748087 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.748048    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.748087 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.748057    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.748087 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.748068    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmrvw\" (UniqueName: \"kubernetes.io/projected/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-kube-api-access-fmrvw\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.748087 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.748083    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:12.748087 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:12.748091    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e617d1c-9103-4aa6-a43e-aaa84c55ae57-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:13.024827 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.024724    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerID="779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3" exitCode=0
Sep 09 21:04:13.024827 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.024752    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerID="b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976" exitCode=0
Sep 09 21:04:13.024827 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.024783    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"
Sep 09 21:04:13.024827 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.024788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" event={"ID":"2e617d1c-9103-4aa6-a43e-aaa84c55ae57","Type":"ContainerDied","Data":"779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3"}
Sep 09 21:04:13.025152 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.024837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" event={"ID":"2e617d1c-9103-4aa6-a43e-aaa84c55ae57","Type":"ContainerDied","Data":"b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976"}
Sep 09 21:04:13.025152 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.024859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx" event={"ID":"2e617d1c-9103-4aa6-a43e-aaa84c55ae57","Type":"ContainerDied","Data":"f3b8795f180b4679e976a71d9f24449c355778c8a7aa2309fffbb4a31174c0b4"}
Sep 09 21:04:13.025152 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.024883    2578 scope.go:117] "RemoveContainer" containerID="779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3"
Sep 09 21:04:13.026607 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.026574    2578 generic.go:356] "Generic (PLEG): container finished" podID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerID="8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7" exitCode=0
Sep 09 21:04:13.026731 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.026638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" event={"ID":"41d5f3e6-67f7-432f-825a-284ee20acb77","Type":"ContainerDied","Data":"8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7"}
Sep 09 21:04:13.026731 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.026646    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"
Sep 09 21:04:13.026731 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.026670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2" event={"ID":"41d5f3e6-67f7-432f-825a-284ee20acb77","Type":"ContainerDied","Data":"c3faed9a3fc5ffea089882ed32cf07e64be47a50c453b79b7243fabf57fcb945"}
Sep 09 21:04:13.033663 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.033645    2578 scope.go:117] "RemoveContainer" containerID="2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7"
Sep 09 21:04:13.042591 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.042576    2578 scope.go:117] "RemoveContainer" containerID="b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976"
Sep 09 21:04:13.050185 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.050163    2578 scope.go:117] "RemoveContainer" containerID="779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3"
Sep 09 21:04:13.050514 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:04:13.050487    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3\": container with ID starting with 779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3 not found: ID does not exist" containerID="779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3"
Sep 09 21:04:13.050675 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.050642    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3"} err="failed to get container status \"779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3\": rpc error: code = NotFound desc = could not find container \"779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3\": container with ID starting with 779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3 not found: ID does not exist"
Sep 09 21:04:13.050675 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.050669    2578 scope.go:117] "RemoveContainer" containerID="2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7"
Sep 09 21:04:13.050945 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:04:13.050928    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7\": container with ID starting with 2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7 not found: ID does not exist" containerID="2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7"
Sep 09 21:04:13.050995 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.050949    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7"} err="failed to get container status \"2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7\": rpc error: code = NotFound desc = could not find container \"2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7\": container with ID starting with 2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7 not found: ID does not exist"
Sep 09 21:04:13.050995 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.050962    2578 scope.go:117] "RemoveContainer" containerID="b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976"
Sep 09 21:04:13.051196 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:04:13.051176    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976\": container with ID starting with b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976 not found: ID does not exist" containerID="b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976"
Sep 09 21:04:13.051275 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.051204    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976"} err="failed to get container status \"b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976\": rpc error: code = NotFound desc = could not find container \"b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976\": container with ID starting with b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976 not found: ID does not exist"
Sep 09 21:04:13.051275 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.051228    2578 scope.go:117] "RemoveContainer" containerID="779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3"
Sep 09 21:04:13.051560 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.051531    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3"} err="failed to get container status \"779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3\": rpc error: code = NotFound desc = could not find container \"779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3\": container with ID starting with 779ce97263fc1de843953a17c1cda368871244af04a68e472c22c0dea13badb3 not found: ID does not exist"
Sep 09 21:04:13.051560 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.051555    2578 scope.go:117] "RemoveContainer" containerID="2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7"
Sep 09 21:04:13.051923 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.051899    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7"} err="failed to get container status \"2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7\": rpc error: code = NotFound desc = could not find container \"2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7\": container with ID starting with 2da8dd0c4d25c7af2de023adf9a59283bdece9313274a8dd762aa8a9837fb7e7 not found: ID does not exist"
Sep 09 21:04:13.052015 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.051924    2578 scope.go:117] "RemoveContainer" containerID="b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976"
Sep 09 21:04:13.052325 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.052282    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976"} err="failed to get container status \"b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976\": rpc error: code = NotFound desc = could not find container \"b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976\": container with ID starting with b205fd8f4ad5d66c4eca8f7bd5f2a078a3d3da04faf65f72900bbecab16f9976 not found: ID does not exist"
Sep 09 21:04:13.052408 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.052326    2578 scope.go:117] "RemoveContainer" containerID="8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7"
Sep 09 21:04:13.053234 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.053214    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"]
Sep 09 21:04:13.057360 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.057341    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-556656f58d-4svlx"]
Sep 09 21:04:13.060754 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.060736    2578 scope.go:117] "RemoveContainer" containerID="8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29"
Sep 09 21:04:13.068078 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.068057    2578 scope.go:117] "RemoveContainer" containerID="8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7"
Sep 09 21:04:13.068376 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:04:13.068324    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7\": container with ID starting with 8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7 not found: ID does not exist" containerID="8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7"
Sep 09 21:04:13.068376 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.068355    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7"} err="failed to get container status \"8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7\": rpc error: code = NotFound desc = could not find container \"8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7\": container with ID starting with 8f868d9030775076a685f81f114d75daee617830a9fbb19de321cd38045c49d7 not found: ID does not exist"
Sep 09 21:04:13.068376 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.068373    2578 scope.go:117] "RemoveContainer" containerID="8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29"
Sep 09 21:04:13.068624 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:04:13.068606    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29\": container with ID starting with 8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29 not found: ID does not exist" containerID="8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29"
Sep 09 21:04:13.068674 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.068630    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29"} err="failed to get container status \"8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29\": rpc error: code = NotFound desc = could not find container \"8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29\": container with ID starting with 8f7525d20f55b39a51c74a34593fe030f7918fbcd0efc1a9fa818854978c3c29 not found: ID does not exist"
Sep 09 21:04:13.071739 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.071714    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"]
Sep 09 21:04:13.073947 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.073926    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7fc8868c95-596n2"]
Sep 09 21:04:13.896422 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.896390    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" path="/var/lib/kubelet/pods/2e617d1c-9103-4aa6-a43e-aaa84c55ae57/volumes"
Sep 09 21:04:13.896916 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:13.896902    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" path="/var/lib/kubelet/pods/41d5f3e6-67f7-432f-825a-284ee20acb77/volumes"
Sep 09 21:04:22.610885 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:22.610850    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:24.717830 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:24.717793    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:46.555561 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:46.555525    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:46.650707 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:46.650666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:51.510026 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:51.509985    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"]
Sep 09 21:04:51.510461 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:51.510202    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy" containerID="cri-o://06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e" gracePeriod=30
Sep 09 21:04:51.620000 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:51.619957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:52.167946 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:52.167906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:52.197524 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:52.197496    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 21:04:52.215227 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:52.215198    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ccdvx_375948f8-eecf-4ce7-a0ab-195b78a6d99f/ovn-acl-logging/0.log"
Sep 09 21:04:52.432249 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:52.432165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:56.767420 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.767392    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:04:56.821667 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821584    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-token\") pod \"6fb073d3-8a01-409c-adb3-8163bedb0db3\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") "
Sep 09 21:04:56.821667 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-podinfo\") pod \"6fb073d3-8a01-409c-adb3-8163bedb0db3\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") "
Sep 09 21:04:56.821890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821678    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-socket\") pod \"6fb073d3-8a01-409c-adb3-8163bedb0db3\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") "
Sep 09 21:04:56.821890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-certs\") pod \"6fb073d3-8a01-409c-adb3-8163bedb0db3\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") "
Sep 09 21:04:56.821890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821750    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97954\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-kube-api-access-97954\") pod \"6fb073d3-8a01-409c-adb3-8163bedb0db3\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") "
Sep 09 21:04:56.821890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821782    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-data\") pod \"6fb073d3-8a01-409c-adb3-8163bedb0db3\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") "
Sep 09 21:04:56.821890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821825    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb073d3-8a01-409c-adb3-8163bedb0db3-istiod-ca-cert\") pod \"6fb073d3-8a01-409c-adb3-8163bedb0db3\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") "
Sep 09 21:04:56.821890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-envoy\") pod \"6fb073d3-8a01-409c-adb3-8163bedb0db3\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") "
Sep 09 21:04:56.821890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-credential-socket\") pod \"6fb073d3-8a01-409c-adb3-8163bedb0db3\" (UID: \"6fb073d3-8a01-409c-adb3-8163bedb0db3\") "
Sep 09 21:04:56.822263 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.821988    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6fb073d3-8a01-409c-adb3-8163bedb0db3" (UID: "6fb073d3-8a01-409c-adb3-8163bedb0db3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:56.822263 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.822021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6fb073d3-8a01-409c-adb3-8163bedb0db3" (UID: "6fb073d3-8a01-409c-adb3-8163bedb0db3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:56.822263 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.822225    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-data" (OuterVolumeSpecName: "istio-data") pod "6fb073d3-8a01-409c-adb3-8163bedb0db3" (UID: "6fb073d3-8a01-409c-adb3-8163bedb0db3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:56.822263 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.822234    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6fb073d3-8a01-409c-adb3-8163bedb0db3" (UID: "6fb073d3-8a01-409c-adb3-8163bedb0db3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:56.822510 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.822481    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb073d3-8a01-409c-adb3-8163bedb0db3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6fb073d3-8a01-409c-adb3-8163bedb0db3" (UID: "6fb073d3-8a01-409c-adb3-8163bedb0db3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:56.822572 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.822547    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-credential-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:56.822647 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.822570    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-socket\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:56.822647 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.822591    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-workload-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:56.822647 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.822605    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-data\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:56.824467 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.824428    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6fb073d3-8a01-409c-adb3-8163bedb0db3" (UID: "6fb073d3-8a01-409c-adb3-8163bedb0db3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:04:56.824467 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.824458    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6fb073d3-8a01-409c-adb3-8163bedb0db3" (UID: "6fb073d3-8a01-409c-adb3-8163bedb0db3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:56.824695 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.824538    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-token" (OuterVolumeSpecName: "istio-token") pod "6fb073d3-8a01-409c-adb3-8163bedb0db3" (UID: "6fb073d3-8a01-409c-adb3-8163bedb0db3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:56.824695 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.824555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-kube-api-access-97954" (OuterVolumeSpecName: "kube-api-access-97954") pod "6fb073d3-8a01-409c-adb3-8163bedb0db3" (UID: "6fb073d3-8a01-409c-adb3-8163bedb0db3"). InnerVolumeSpecName "kube-api-access-97954". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:56.923670 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.923629    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-token\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:56.923670 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.923660    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-podinfo\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:56.923670 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.923669    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97954\" (UniqueName: \"kubernetes.io/projected/6fb073d3-8a01-409c-adb3-8163bedb0db3-kube-api-access-97954\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:56.923670 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.923679    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fb073d3-8a01-409c-adb3-8163bedb0db3-istiod-ca-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:56.924128 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:56.923687    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fb073d3-8a01-409c-adb3-8163bedb0db3-istio-envoy\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:04:57.231625 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.231589    2578 generic.go:356] "Generic (PLEG): container finished" podID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerID="06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e" exitCode=0
Sep 09 21:04:57.231826 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.231638    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"
Sep 09 21:04:57.231826 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.231642    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" event={"ID":"6fb073d3-8a01-409c-adb3-8163bedb0db3","Type":"ContainerDied","Data":"06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e"}
Sep 09 21:04:57.231826 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.231783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb" event={"ID":"6fb073d3-8a01-409c-adb3-8163bedb0db3","Type":"ContainerDied","Data":"cd6ad05dc51be1fc87fbfde01aa9861d666236830bc030b91e9d9ec406231987"}
Sep 09 21:04:57.231826 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.231822    2578 scope.go:117] "RemoveContainer" containerID="06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e"
Sep 09 21:04:57.240828 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.240812    2578 scope.go:117] "RemoveContainer" containerID="9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b"
Sep 09 21:04:57.248609 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.248589    2578 scope.go:117] "RemoveContainer" containerID="06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e"
Sep 09 21:04:57.248904 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:04:57.248886    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e\": container with ID starting with 06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e not found: ID does not exist" containerID="06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e"
Sep 09 21:04:57.248985 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.248918    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e"} err="failed to get container status \"06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e\": rpc error: code = NotFound desc = could not find container \"06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e\": container with ID starting with 06c901ecdd92bd4fe5f434535ef272c0c44c21765086f6ab701b04c7c2a23b9e not found: ID does not exist"
Sep 09 21:04:57.248985 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.248943    2578 scope.go:117] "RemoveContainer" containerID="9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b"
Sep 09 21:04:57.249221 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:04:57.249202    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b\": container with ID starting with 9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b not found: ID does not exist" containerID="9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b"
Sep 09 21:04:57.249265 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.249230    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b"} err="failed to get container status \"9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b\": rpc error: code = NotFound desc = could not find container \"9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b\": container with ID starting with 9a543b138b7b61b10b06ec272dc2ad59c9fc098f3604765c83e7c772f76b554b not found: ID does not exist"
Sep 09 21:04:57.254230 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.254209    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"]
Sep 09 21:04:57.263834 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.263804    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-cpfwb"]
Sep 09 21:04:57.896133 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:57.896084    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" path="/var/lib/kubelet/pods/6fb073d3-8a01-409c-adb3-8163bedb0db3/volumes"
Sep 09 21:04:58.344082 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:04:58.344054    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:12.739407 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:12.739372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:12.878093 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:12.878062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:13.531230 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:13.531200    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:15.015147 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:15.015116    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:18.971761 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:18.971726    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:21.065852 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:21.065821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:24.781661 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:24.781615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:24.855769 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:24.855738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:31.731407 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731365    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"]
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731726    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731738    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731746    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="llm-d-routing-sidecar"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731752    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="llm-d-routing-sidecar"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731758    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731763    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731775    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731780    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731789    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="storage-initializer"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731795    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="storage-initializer"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731802    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731808    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731816    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="storage-initializer"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731821    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="storage-initializer"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731879    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="llm-d-routing-sidecar"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731886    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731894    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fb073d3-8a01-409c-adb3-8163bedb0db3" containerName="istio-proxy"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731900    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e617d1c-9103-4aa6-a43e-aaa84c55ae57" containerName="main"
Sep 09 21:05:31.731890 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.731904    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d5f3e6-67f7-432f-825a-284ee20acb77" containerName="main"
Sep 09 21:05:31.734266 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.734245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.737987 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.737963    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 09 21:05:31.738134 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.737965    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 09 21:05:31.738226 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.737969    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 09 21:05:31.738280 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.738005    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-dfg67\""
Sep 09 21:05:31.743127 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.743106    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"]
Sep 09 21:05:31.768042 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.768013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:31.823738 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.823703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.823738 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.823740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.823984 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.823813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.823984 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.823847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.823984 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.823888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.823984 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.823907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5s6c\" (UniqueName: \"kubernetes.io/projected/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-kube-api-access-s5s6c\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.924549 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.924510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5s6c\" (UniqueName: \"kubernetes.io/projected/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-kube-api-access-s5s6c\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.924772 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.924585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.924772 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.924614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.924772 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.924670    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.925000 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.924970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.925071 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.925026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.925071 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.925062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.925182 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.925065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.925284 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.925259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.927012 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.926988    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.927478 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.927457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:31.933017 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:31.932994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5s6c\" (UniqueName: \"kubernetes.io/projected/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-kube-api-access-s5s6c\") pod \"rolling-upgrade-test-kserve-74b79d7f77-nbs49\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:32.036480 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:32.036393    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:32.045274 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:32.045248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:32.316630 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:32.316555    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"]
Sep 09 21:05:32.319068 ip-10-0-138-141 kubenswrapper[2578]: W0909 21:05:32.319032    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1c8f1b_a483_4a96_8d6e_85ad5330ef5d.slice/crio-7df4bd1f0496274b6c02dbef44ccf1e816602c1281b573b5755de9d815db68ba WatchSource:0}: Error finding container 7df4bd1f0496274b6c02dbef44ccf1e816602c1281b573b5755de9d815db68ba: Status 404 returned error can't find the container with id 7df4bd1f0496274b6c02dbef44ccf1e816602c1281b573b5755de9d815db68ba
Sep 09 21:05:32.386533 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:32.386507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49" event={"ID":"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d","Type":"ContainerStarted","Data":"7df4bd1f0496274b6c02dbef44ccf1e816602c1281b573b5755de9d815db68ba"}
Sep 09 21:05:33.392638 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:33.392604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49" event={"ID":"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d","Type":"ContainerStarted","Data":"9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e"}
Sep 09 21:05:33.413444 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:33.413391    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49" podStartSLOduration=2.41337621 podStartE2EDuration="2.41337621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:33.41145925 +0000 UTC m=+4242.000112929" watchObservedRunningTime="2026-09-09 21:05:33.41337621 +0000 UTC m=+4242.002029867"
Sep 09 21:05:34.046378 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:34.046302    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:34.964032 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:34.963996    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:35.676469 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:35.676416    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:35.913463 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:35.913428    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:42.046276 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:42.046235    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:42.058756 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:42.058718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:42.447793 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:42.447762    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:05:47.217775 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:05:47.217746    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:06.471895 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.471859    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"]
Sep 09 21:06:06.474970 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.474952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.485421 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.485393    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"]
Sep 09 21:06:06.619587 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.619540    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.619587 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.619588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.619818 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.619656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.619818 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.619725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.619818 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.619746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7c4d\" (UniqueName: \"kubernetes.io/projected/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-kube-api-access-s7c4d\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.619818 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.619792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.720238 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.720195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.720238 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.720235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.720531 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.720255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.720531 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.720281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.720531 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.720334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.720531 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.720352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7c4d\" (UniqueName: \"kubernetes.io/projected/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-kube-api-access-s7c4d\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.720754 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.720713    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.720799 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.720747    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.720799 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.720759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.722787 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.722718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.723081 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.723059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.729582 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.729558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7c4d\" (UniqueName: \"kubernetes.io/projected/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-kube-api-access-s7c4d\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-t65wg\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:06.784969 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:06.784934    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:07.063072 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:07.062980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"]
Sep 09 21:06:07.066970 ip-10-0-138-141 kubenswrapper[2578]: W0909 21:06:07.066939    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1a19a2_9b3a_4eac_b1b2_456a61f4db2f.slice/crio-8cf47d732a72f173f25600ca41329d300eede5648f3c2ec30dee3e4421480ac8 WatchSource:0}: Error finding container 8cf47d732a72f173f25600ca41329d300eede5648f3c2ec30dee3e4421480ac8: Status 404 returned error can't find the container with id 8cf47d732a72f173f25600ca41329d300eede5648f3c2ec30dee3e4421480ac8
Sep 09 21:06:07.542831 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:07.542791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg" event={"ID":"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f","Type":"ContainerStarted","Data":"c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7"}
Sep 09 21:06:07.543331 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:07.542853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg" event={"ID":"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f","Type":"ContainerStarted","Data":"8cf47d732a72f173f25600ca41329d300eede5648f3c2ec30dee3e4421480ac8"}
Sep 09 21:06:07.564556 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:07.564497    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg" podStartSLOduration=1.5644789019999998 podStartE2EDuration="1.564478902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:07.562953075 +0000 UTC m=+4276.151606731" watchObservedRunningTime="2026-09-09 21:06:07.564478902 +0000 UTC m=+4276.153132560"
Sep 09 21:06:07.785963 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:07.785929    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:10.864361 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:10.864284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:10.894589 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:10.894555    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"]
Sep 09 21:06:10.899764 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:10.899724    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"]
Sep 09 21:06:10.903965 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:06:10.903930    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:06:10.903965 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:06:10.903965 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 21:06:10.903965 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e" cmd=["/bin/sleep","15"]
Sep 09 21:06:10.904333 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:10.903979    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49" message="" err=<
Sep 09 21:06:10.904333 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:06:10.904333 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 21:06:10.904333 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 21:06:10.904333 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:06:10.903992    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:06:10.904333 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:06:10.904333 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 21:06:10.904333 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49" podUID="ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" containerName="main" containerID="cri-o://9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e"
Sep 09 21:06:10.904333 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:10.904028    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49" podUID="ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" containerName="main" containerID="cri-o://9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e" gracePeriod=60
Sep 09 21:06:10.908843 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:06:10.908809    2578 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 21:06:10.908843 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:06:10.908843 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 21:06:10.908843 ip-10-0-138-141 kubenswrapper[2578]:  > containerID="c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7" cmd=["/bin/sleep","15"]
Sep 09 21:06:10.909044 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:10.908859    2578 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg" message="" err=<
Sep 09 21:06:10.909044 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:06:10.909044 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 21:06:10.909044 ip-10-0-138-141 kubenswrapper[2578]:  >
Sep 09 21:06:10.909044 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:06:10.908871    2578 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 21:06:10.909044 ip-10-0-138-141 kubenswrapper[2578]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 21:06:10.909044 ip-10-0-138-141 kubenswrapper[2578]:         , stdout: , stderr: , exit code -1
Sep 09 21:06:10.909044 ip-10-0-138-141 kubenswrapper[2578]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg" podUID="6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" containerName="main" containerID="cri-o://c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7"
Sep 09 21:06:10.909044 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:10.908905    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg" podUID="6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" containerName="main" containerID="cri-o://c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7" gracePeriod=60
Sep 09 21:06:10.959197 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:10.959161    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:11.179188 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.179165    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:11.182207 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.182187    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:06:11.260031 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.259992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tls-certs\") pod \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") "
Sep 09 21:06:11.260226 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260038    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tmp-dir\") pod \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") "
Sep 09 21:06:11.260226 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260065    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-home\") pod \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") "
Sep 09 21:06:11.260226 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-home\") pod \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") "
Sep 09 21:06:11.260226 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-dshm\") pod \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") "
Sep 09 21:06:11.260226 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-model-cache\") pod \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") "
Sep 09 21:06:11.260226 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-dshm\") pod \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") "
Sep 09 21:06:11.260586 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260232    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5s6c\" (UniqueName: \"kubernetes.io/projected/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-kube-api-access-s5s6c\") pod \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") "
Sep 09 21:06:11.260586 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260289    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-model-cache\") pod \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") "
Sep 09 21:06:11.260586 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260353    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7c4d\" (UniqueName: \"kubernetes.io/projected/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-kube-api-access-s7c4d\") pod \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") "
Sep 09 21:06:11.260586 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260377    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" (UID: "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:11.260586 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260389    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" (UID: "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:11.260586 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260404    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-home" (OuterVolumeSpecName: "home") pod "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" (UID: "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:11.260586 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260404    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tls-certs\") pod \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\" (UID: \"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d\") "
Sep 09 21:06:11.260586 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-home" (OuterVolumeSpecName: "home") pod "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" (UID: "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:11.260586 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260460    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tmp-dir\") pod \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\" (UID: \"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f\") "
Sep 09 21:06:11.261063 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260705    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" (UID: "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:11.261063 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260710    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.261063 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260771    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.261063 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260788    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.261063 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260801    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.261063 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.260798    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" (UID: "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:11.262591 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.262551    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" (UID: "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:11.262832 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.262787    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" (UID: "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:11.262963 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.262870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-dshm" (OuterVolumeSpecName: "dshm") pod "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" (UID: "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:11.262963 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.262871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-dshm" (OuterVolumeSpecName: "dshm") pod "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" (UID: "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:11.263116 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.263098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-kube-api-access-s7c4d" (OuterVolumeSpecName: "kube-api-access-s7c4d") pod "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" (UID: "6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f"). InnerVolumeSpecName "kube-api-access-s7c4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:11.263176 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.263122    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-kube-api-access-s5s6c" (OuterVolumeSpecName: "kube-api-access-s5s6c") pod "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" (UID: "ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d"). InnerVolumeSpecName "kube-api-access-s5s6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:11.362067 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.362031    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.362067 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.362059    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7c4d\" (UniqueName: \"kubernetes.io/projected/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-kube-api-access-s7c4d\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.362067 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.362069    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.362296 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.362080    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.362296 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.362088    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.362296 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.362096    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.362296 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.362103    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.362296 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.362112    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5s6c\" (UniqueName: \"kubernetes.io/projected/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d-kube-api-access-s5s6c\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:06:11.561218 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.561118    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" containerID="9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e" exitCode=0
Sep 09 21:06:11.561218 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.561162    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"
Sep 09 21:06:11.561218 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.561206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49" event={"ID":"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d","Type":"ContainerDied","Data":"9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e"}
Sep 09 21:06:11.561552 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.561276    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49" event={"ID":"ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d","Type":"ContainerDied","Data":"7df4bd1f0496274b6c02dbef44ccf1e816602c1281b573b5755de9d815db68ba"}
Sep 09 21:06:11.561552 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.561330    2578 scope.go:117] "RemoveContainer" containerID="9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e"
Sep 09 21:06:11.562635 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.562615    2578 generic.go:356] "Generic (PLEG): container finished" podID="6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" containerID="c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7" exitCode=0
Sep 09 21:06:11.562697 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.562658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg" event={"ID":"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f","Type":"ContainerDied","Data":"c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7"}
Sep 09 21:06:11.562746 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.562704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg" event={"ID":"6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f","Type":"ContainerDied","Data":"8cf47d732a72f173f25600ca41329d300eede5648f3c2ec30dee3e4421480ac8"}
Sep 09 21:06:11.562746 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.562663    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"
Sep 09 21:06:11.570893 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.570874    2578 scope.go:117] "RemoveContainer" containerID="9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e"
Sep 09 21:06:11.571255 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:06:11.571228    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e\": container with ID starting with 9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e not found: ID does not exist" containerID="9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e"
Sep 09 21:06:11.571335 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.571268    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e"} err="failed to get container status \"9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e\": rpc error: code = NotFound desc = could not find container \"9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e\": container with ID starting with 9687b25d2b06ac58f4bb0082f3b4f93d35f76fd4b946b0fca165b78965da277e not found: ID does not exist"
Sep 09 21:06:11.571335 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.571294    2578 scope.go:117] "RemoveContainer" containerID="c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7"
Sep 09 21:06:11.579637 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.579619    2578 scope.go:117] "RemoveContainer" containerID="c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7"
Sep 09 21:06:11.579865 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:06:11.579846    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7\": container with ID starting with c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7 not found: ID does not exist" containerID="c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7"
Sep 09 21:06:11.579927 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.579876    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7"} err="failed to get container status \"c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7\": rpc error: code = NotFound desc = could not find container \"c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7\": container with ID starting with c809f354a6ccedeae49c25b22ef534a8c70e66cc1bb69cd1fd64e20a2b5eb8e7 not found: ID does not exist"
Sep 09 21:06:11.600687 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.600660    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"]
Sep 09 21:06:11.605253 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.605230    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-t65wg"]
Sep 09 21:06:11.640402 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.640371    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"]
Sep 09 21:06:11.641641 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.641619    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-nbs49"]
Sep 09 21:06:11.895943 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.895903    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" path="/var/lib/kubelet/pods/6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f/volumes"
Sep 09 21:06:11.896355 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:11.896285    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" path="/var/lib/kubelet/pods/ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d/volumes"
Sep 09 21:06:18.557141 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:18.557099    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:58.613718 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:06:58.613678    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:02.610038 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.610003    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"]
Sep 09 21:07:02.610446 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.610326    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" containerName="main"
Sep 09 21:07:02.610446 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.610340    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" containerName="main"
Sep 09 21:07:02.610446 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.610360    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" containerName="main"
Sep 09 21:07:02.610446 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.610366    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" containerName="main"
Sep 09 21:07:02.610446 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.610417    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f1a19a2-9b3a-4eac-b1b2-456a61f4db2f" containerName="main"
Sep 09 21:07:02.610446 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.610426    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee1c8f1b-a483-4a96-8d6e-85ad5330ef5d" containerName="main"
Sep 09 21:07:02.614492 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.614474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.617356 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.617331    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 09 21:07:02.617674 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.617654    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4wf97\""
Sep 09 21:07:02.617795 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.617692    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 09 21:07:02.617795 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.617700    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 09 21:07:02.617795 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.617743    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:07:02.618443 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.618422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 09 21:07:02.628394 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.628374    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"]
Sep 09 21:07:02.679867 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.679833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-tmp-dir\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.680013 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.679876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af29f567-06e7-4bde-979c-b13893c0b94c-tls-certs\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.680013 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.679899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-home\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.680013 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.679980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-kserve-provision-location\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.680139 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.680023    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af29f567-06e7-4bde-979c-b13893c0b94c-cabundle-cert\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.680139 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.680046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-model-cache\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.680139 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.680064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-dshm\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.680139 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.680084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx9gs\" (UniqueName: \"kubernetes.io/projected/af29f567-06e7-4bde-979c-b13893c0b94c-kube-api-access-cx9gs\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781251 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-kserve-provision-location\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781432 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af29f567-06e7-4bde-979c-b13893c0b94c-cabundle-cert\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781432 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-model-cache\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781432 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-dshm\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781432 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx9gs\" (UniqueName: \"kubernetes.io/projected/af29f567-06e7-4bde-979c-b13893c0b94c-kube-api-access-cx9gs\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781432 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781411    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-tmp-dir\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781693 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af29f567-06e7-4bde-979c-b13893c0b94c-tls-certs\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781693 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-home\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781790 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-kserve-provision-location\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.781969 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.781937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-tmp-dir\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.782116 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.782001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-home\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.782116 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.782021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-model-cache\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.782248 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.782209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af29f567-06e7-4bde-979c-b13893c0b94c-cabundle-cert\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.783673 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.783645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-dshm\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.783965 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.783948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af29f567-06e7-4bde-979c-b13893c0b94c-tls-certs\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.790132 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.790107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx9gs\" (UniqueName: \"kubernetes.io/projected/af29f567-06e7-4bde-979c-b13893c0b94c-kube-api-access-cx9gs\") pod \"prestop-hook-test-kserve-847f84995d-klxkn\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:02.925483 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:02.925389    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:03.206390 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:03.206359    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"]
Sep 09 21:07:03.208097 ip-10-0-138-141 kubenswrapper[2578]: W0909 21:07:03.208071    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf29f567_06e7_4bde_979c_b13893c0b94c.slice/crio-1a151ca1e4ff8481b45d4f0ce8425b3a76ff24aca0027549d9b1146d489a936f WatchSource:0}: Error finding container 1a151ca1e4ff8481b45d4f0ce8425b3a76ff24aca0027549d9b1146d489a936f: Status 404 returned error can't find the container with id 1a151ca1e4ff8481b45d4f0ce8425b3a76ff24aca0027549d9b1146d489a936f
Sep 09 21:07:03.795801 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:03.795765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" event={"ID":"af29f567-06e7-4bde-979c-b13893c0b94c","Type":"ContainerStarted","Data":"35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec"}
Sep 09 21:07:03.796180 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:03.795818    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" event={"ID":"af29f567-06e7-4bde-979c-b13893c0b94c","Type":"ContainerStarted","Data":"1a151ca1e4ff8481b45d4f0ce8425b3a76ff24aca0027549d9b1146d489a936f"}
Sep 09 21:07:05.417470 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:05.417415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:05.805592 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:05.805497    2578 generic.go:356] "Generic (PLEG): container finished" podID="af29f567-06e7-4bde-979c-b13893c0b94c" containerID="35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec" exitCode=0
Sep 09 21:07:05.805592 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:05.805567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" event={"ID":"af29f567-06e7-4bde-979c-b13893c0b94c","Type":"ContainerDied","Data":"35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec"}
Sep 09 21:07:06.811738 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:06.811701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" event={"ID":"af29f567-06e7-4bde-979c-b13893c0b94c","Type":"ContainerStarted","Data":"c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47"}
Sep 09 21:07:06.841734 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:06.841677    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" podStartSLOduration=2.719855784 podStartE2EDuration="4.841657654s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.12180187s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:06.841224142 +0000 UTC m=+4335.429877801" watchObservedRunningTime="2026-09-09 21:07:06.841657654 +0000 UTC m=+4335.430311338"
Sep 09 21:07:06.925901 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:06.925868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:11.491126 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:11.491090    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:11.642411 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:11.642367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:12.925941 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:12.925900    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:07:12.927406 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:12.927374    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:07:22.926748 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:22.926698    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:07:32.926663 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:32.926614    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:07:42.926892 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:42.926840    2578 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 09 21:07:44.096427 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:44.096388    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"]
Sep 09 21:07:44.101161 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:44.101114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:44.201650 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:44.201595    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:59.122847 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:07:59.122785    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="main" containerID="cri-o://c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47" gracePeriod=45
Sep 09 21:08:01.061640 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.061610    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-847f84995d-klxkn_af29f567-06e7-4bde-979c-b13893c0b94c/main/0.log"
Sep 09 21:08:01.064414 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.064394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:08:01.072501 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.072474    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-847f84995d-klxkn_af29f567-06e7-4bde-979c-b13893c0b94c/main/0.log"
Sep 09 21:08:01.072953 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.072925    2578 generic.go:356] "Generic (PLEG): container finished" podID="af29f567-06e7-4bde-979c-b13893c0b94c" containerID="c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47" exitCode=130
Sep 09 21:08:01.073086 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.072988    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"
Sep 09 21:08:01.073086 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.073004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" event={"ID":"af29f567-06e7-4bde-979c-b13893c0b94c","Type":"ContainerDied","Data":"c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47"}
Sep 09 21:08:01.073086 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.073071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn" event={"ID":"af29f567-06e7-4bde-979c-b13893c0b94c","Type":"ContainerDied","Data":"1a151ca1e4ff8481b45d4f0ce8425b3a76ff24aca0027549d9b1146d489a936f"}
Sep 09 21:08:01.073209 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.073103    2578 scope.go:117] "RemoveContainer" containerID="c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47"
Sep 09 21:08:01.081511 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.081487    2578 scope.go:117] "RemoveContainer" containerID="35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec"
Sep 09 21:08:01.090137 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.090112    2578 scope.go:117] "RemoveContainer" containerID="c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47"
Sep 09 21:08:01.090502 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:08:01.090480    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47\": container with ID starting with c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47 not found: ID does not exist" containerID="c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47"
Sep 09 21:08:01.090587 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.090511    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47"} err="failed to get container status \"c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47\": rpc error: code = NotFound desc = could not find container \"c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47\": container with ID starting with c0806b4ad9f9293117d5f93389dce43ce2b6968d3f9455843874dc42fe1f7d47 not found: ID does not exist"
Sep 09 21:08:01.090587 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.090532    2578 scope.go:117] "RemoveContainer" containerID="35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec"
Sep 09 21:08:01.090837 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:08:01.090817    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec\": container with ID starting with 35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec not found: ID does not exist" containerID="35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec"
Sep 09 21:08:01.090893 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.090843    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec"} err="failed to get container status \"35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec\": rpc error: code = NotFound desc = could not find container \"35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec\": container with ID starting with 35f9dba0e517503653ecdcf5f68a16096652ab81cf1ccd7422d6912cb17434ec not found: ID does not exist"
Sep 09 21:08:01.202071 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.201974    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-model-cache\") pod \"af29f567-06e7-4bde-979c-b13893c0b94c\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") "
Sep 09 21:08:01.202071 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-home\") pod \"af29f567-06e7-4bde-979c-b13893c0b94c\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") "
Sep 09 21:08:01.202071 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202056    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af29f567-06e7-4bde-979c-b13893c0b94c-cabundle-cert\") pod \"af29f567-06e7-4bde-979c-b13893c0b94c\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") "
Sep 09 21:08:01.202416 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-tmp-dir\") pod \"af29f567-06e7-4bde-979c-b13893c0b94c\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") "
Sep 09 21:08:01.202416 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-kserve-provision-location\") pod \"af29f567-06e7-4bde-979c-b13893c0b94c\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") "
Sep 09 21:08:01.202416 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af29f567-06e7-4bde-979c-b13893c0b94c-tls-certs\") pod \"af29f567-06e7-4bde-979c-b13893c0b94c\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") "
Sep 09 21:08:01.202416 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202192    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx9gs\" (UniqueName: \"kubernetes.io/projected/af29f567-06e7-4bde-979c-b13893c0b94c-kube-api-access-cx9gs\") pod \"af29f567-06e7-4bde-979c-b13893c0b94c\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") "
Sep 09 21:08:01.202416 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202230    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-dshm\") pod \"af29f567-06e7-4bde-979c-b13893c0b94c\" (UID: \"af29f567-06e7-4bde-979c-b13893c0b94c\") "
Sep 09 21:08:01.202416 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-model-cache" (OuterVolumeSpecName: "model-cache") pod "af29f567-06e7-4bde-979c-b13893c0b94c" (UID: "af29f567-06e7-4bde-979c-b13893c0b94c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:01.202729 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202461    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-home" (OuterVolumeSpecName: "home") pod "af29f567-06e7-4bde-979c-b13893c0b94c" (UID: "af29f567-06e7-4bde-979c-b13893c0b94c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:01.202729 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af29f567-06e7-4bde-979c-b13893c0b94c" (UID: "af29f567-06e7-4bde-979c-b13893c0b94c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:01.202729 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202531    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-model-cache\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:08:01.202729 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202552    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-home\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:08:01.202930 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.202732    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af29f567-06e7-4bde-979c-b13893c0b94c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "af29f567-06e7-4bde-979c-b13893c0b94c" (UID: "af29f567-06e7-4bde-979c-b13893c0b94c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:01.204665 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.204631    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af29f567-06e7-4bde-979c-b13893c0b94c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af29f567-06e7-4bde-979c-b13893c0b94c" (UID: "af29f567-06e7-4bde-979c-b13893c0b94c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:01.204779 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.204642    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-dshm" (OuterVolumeSpecName: "dshm") pod "af29f567-06e7-4bde-979c-b13893c0b94c" (UID: "af29f567-06e7-4bde-979c-b13893c0b94c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:01.204779 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.204731    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af29f567-06e7-4bde-979c-b13893c0b94c-kube-api-access-cx9gs" (OuterVolumeSpecName: "kube-api-access-cx9gs") pod "af29f567-06e7-4bde-979c-b13893c0b94c" (UID: "af29f567-06e7-4bde-979c-b13893c0b94c"). InnerVolumeSpecName "kube-api-access-cx9gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:01.227556 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.227523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af29f567-06e7-4bde-979c-b13893c0b94c" (UID: "af29f567-06e7-4bde-979c-b13893c0b94c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:01.303979 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.303944    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af29f567-06e7-4bde-979c-b13893c0b94c-cabundle-cert\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:08:01.304149 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.303992    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-tmp-dir\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:08:01.304149 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.304003    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-kserve-provision-location\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:08:01.304149 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.304014    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af29f567-06e7-4bde-979c-b13893c0b94c-tls-certs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:08:01.304149 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.304023    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx9gs\" (UniqueName: \"kubernetes.io/projected/af29f567-06e7-4bde-979c-b13893c0b94c-kube-api-access-cx9gs\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:08:01.304149 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.304032    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af29f567-06e7-4bde-979c-b13893c0b94c-dshm\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:08:01.399275 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.399242    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"]
Sep 09 21:08:01.402373 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.402345    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-847f84995d-klxkn"]
Sep 09 21:08:01.896699 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:01.896661    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" path="/var/lib/kubelet/pods/af29f567-06e7-4bde-979c-b13893c0b94c/volumes"
Sep 09 21:08:17.586957 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:17.586877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:31.302511 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:31.302477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:32.899510 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.899471    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sx2kw/must-gather-jwks7"]
Sep 09 21:08:32.899929 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.899780    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="main"
Sep 09 21:08:32.899929 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.899791    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="main"
Sep 09 21:08:32.899929 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.899800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="storage-initializer"
Sep 09 21:08:32.899929 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.899806    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="storage-initializer"
Sep 09 21:08:32.899929 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.899863    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="af29f567-06e7-4bde-979c-b13893c0b94c" containerName="main"
Sep 09 21:08:32.902878 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.902845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:08:32.905273 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.905249    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sx2kw\"/\"kube-root-ca.crt\""
Sep 09 21:08:32.906160 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.906124    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sx2kw\"/\"openshift-service-ca.crt\""
Sep 09 21:08:32.906285 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.906200    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sx2kw\"/\"default-dockercfg-rlk6f\""
Sep 09 21:08:32.911496 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.911474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sx2kw/must-gather-jwks7"]
Sep 09 21:08:32.917601 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.917580    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:32.959391 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.959348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e514d9d9-ed69-4be2-8161-084fa14fbea4-must-gather-output\") pod \"must-gather-jwks7\" (UID: \"e514d9d9-ed69-4be2-8161-084fa14fbea4\") " pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:08:32.959579 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.959445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmds8\" (UniqueName: \"kubernetes.io/projected/e514d9d9-ed69-4be2-8161-084fa14fbea4-kube-api-access-tmds8\") pod \"must-gather-jwks7\" (UID: \"e514d9d9-ed69-4be2-8161-084fa14fbea4\") " pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:08:32.995492 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:32.995461    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:33.059935 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:33.059904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e514d9d9-ed69-4be2-8161-084fa14fbea4-must-gather-output\") pod \"must-gather-jwks7\" (UID: \"e514d9d9-ed69-4be2-8161-084fa14fbea4\") " pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:08:33.060137 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:33.059960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmds8\" (UniqueName: \"kubernetes.io/projected/e514d9d9-ed69-4be2-8161-084fa14fbea4-kube-api-access-tmds8\") pod \"must-gather-jwks7\" (UID: \"e514d9d9-ed69-4be2-8161-084fa14fbea4\") " pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:08:33.060301 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:33.060279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e514d9d9-ed69-4be2-8161-084fa14fbea4-must-gather-output\") pod \"must-gather-jwks7\" (UID: \"e514d9d9-ed69-4be2-8161-084fa14fbea4\") " pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:08:33.068717 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:33.068688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmds8\" (UniqueName: \"kubernetes.io/projected/e514d9d9-ed69-4be2-8161-084fa14fbea4-kube-api-access-tmds8\") pod \"must-gather-jwks7\" (UID: \"e514d9d9-ed69-4be2-8161-084fa14fbea4\") " pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:08:33.212184 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:33.212104    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:08:33.483916 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:33.483889    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sx2kw/must-gather-jwks7"]
Sep 09 21:08:33.486603 ip-10-0-138-141 kubenswrapper[2578]: W0909 21:08:33.486573    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode514d9d9_ed69_4be2_8161_084fa14fbea4.slice/crio-4bee157d7cffebc039dfaf4141d3c37514e41ee4fe67168b625a00edcf60dcb4 WatchSource:0}: Error finding container 4bee157d7cffebc039dfaf4141d3c37514e41ee4fe67168b625a00edcf60dcb4: Status 404 returned error can't find the container with id 4bee157d7cffebc039dfaf4141d3c37514e41ee4fe67168b625a00edcf60dcb4
Sep 09 21:08:33.488612 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:33.488593    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:08:34.215652 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:34.215566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sx2kw/must-gather-jwks7" event={"ID":"e514d9d9-ed69-4be2-8161-084fa14fbea4","Type":"ContainerStarted","Data":"4bee157d7cffebc039dfaf4141d3c37514e41ee4fe67168b625a00edcf60dcb4"}
Sep 09 21:08:38.048947 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:38.048907    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:40.248428 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:40.248327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sx2kw/must-gather-jwks7" event={"ID":"e514d9d9-ed69-4be2-8161-084fa14fbea4","Type":"ContainerStarted","Data":"e6c2118feaa90c64ab2d28c46aed023083893bebed6925548af23c9cb3b2967b"}
Sep 09 21:08:40.248428 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:40.248381    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sx2kw/must-gather-jwks7" event={"ID":"e514d9d9-ed69-4be2-8161-084fa14fbea4","Type":"ContainerStarted","Data":"0487ebf40279de3a9aca9d1fe6cf2c662811b6e61cfd942b3631d290909e24df"}
Sep 09 21:08:40.268917 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:40.268858    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sx2kw/must-gather-jwks7" podStartSLOduration=2.099123575 podStartE2EDuration="8.268840424s" totalImagesPullingTime="6.169716849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:08:40.268590848 +0000 UTC m=+4428.857244506" watchObservedRunningTime="2026-09-09 21:08:40.268840424 +0000 UTC m=+4428.857494081"
Sep 09 21:08:44.033973 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:44.033934    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:44.141904 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:44.141867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:56.158226 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:56.158182    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/0.log"
Sep 09 21:08:56.174514 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:56.174482    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/1.log"
Sep 09 21:08:56.915505 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:56.915464    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/0.log"
Sep 09 21:08:56.933379 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:56.933346    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/1.log"
Sep 09 21:08:57.944537 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:57.944501    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-c6hdh_816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4/authorino/0.log"
Sep 09 21:08:58.060551 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:08:58.060522    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9hp8f_cd920ac6-11e4-4c6a-99c1-d48bbf0500ba/kuadrant-console-plugin/0.log"
Sep 09 21:09:03.376724 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:03.376683    2578 generic.go:356] "Generic (PLEG): container finished" podID="e514d9d9-ed69-4be2-8161-084fa14fbea4" containerID="0487ebf40279de3a9aca9d1fe6cf2c662811b6e61cfd942b3631d290909e24df" exitCode=0
Sep 09 21:09:03.377157 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:03.376734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sx2kw/must-gather-jwks7" event={"ID":"e514d9d9-ed69-4be2-8161-084fa14fbea4","Type":"ContainerDied","Data":"0487ebf40279de3a9aca9d1fe6cf2c662811b6e61cfd942b3631d290909e24df"}
Sep 09 21:09:03.377157 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:03.377048    2578 scope.go:117] "RemoveContainer" containerID="0487ebf40279de3a9aca9d1fe6cf2c662811b6e61cfd942b3631d290909e24df"
Sep 09 21:09:03.677636 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:03.677548    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sx2kw_must-gather-jwks7_e514d9d9-ed69-4be2-8161-084fa14fbea4/gather/0.log"
Sep 09 21:09:04.222604 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:04.222569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:04.391415 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:04.391380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:04.482757 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:04.482662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:07.391003 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:07.390967    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2458j_34db695b-0f6e-43c2-b92f-65214c6f28e4/global-pull-secret-syncer/0.log"
Sep 09 21:09:07.537723 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:07.537691    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log"
Sep 09 21:09:07.692169 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:07.692092    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-141.ec2.internal_181a392e75f0abd87761766443396815/haproxy/0.log"
Sep 09 21:09:09.194324 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.194276    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-sx2kw/must-gather-jwks7"]
Sep 09 21:09:09.194868 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.194465    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-sx2kw/must-gather-jwks7" podUID="e514d9d9-ed69-4be2-8161-084fa14fbea4" containerName="copy" containerID="cri-o://e6c2118feaa90c64ab2d28c46aed023083893bebed6925548af23c9cb3b2967b" gracePeriod=2
Sep 09 21:09:09.199975 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.199951    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-sx2kw/must-gather-jwks7"]
Sep 09 21:09:09.406674 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.406648    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sx2kw_must-gather-jwks7_e514d9d9-ed69-4be2-8161-084fa14fbea4/copy/0.log"
Sep 09 21:09:09.407066 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.407035    2578 generic.go:356] "Generic (PLEG): container finished" podID="e514d9d9-ed69-4be2-8161-084fa14fbea4" containerID="e6c2118feaa90c64ab2d28c46aed023083893bebed6925548af23c9cb3b2967b" exitCode=143
Sep 09 21:09:09.430113 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.430095    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sx2kw_must-gather-jwks7_e514d9d9-ed69-4be2-8161-084fa14fbea4/copy/0.log"
Sep 09 21:09:09.431662 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.431646    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:09:09.436597 ip-10-0-138-141 kubenswrapper[2578]: E0909 21:09:09.436574    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jwks7\" is forbidden: User \"system:node:ip-10-0-138-141.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sx2kw\": no relationship found between node 'ip-10-0-138-141.ec2.internal' and this object" podUID="e514d9d9-ed69-4be2-8161-084fa14fbea4" pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:09:09.578043 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.577965    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmds8\" (UniqueName: \"kubernetes.io/projected/e514d9d9-ed69-4be2-8161-084fa14fbea4-kube-api-access-tmds8\") pod \"e514d9d9-ed69-4be2-8161-084fa14fbea4\" (UID: \"e514d9d9-ed69-4be2-8161-084fa14fbea4\") "
Sep 09 21:09:09.578043 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.578033    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e514d9d9-ed69-4be2-8161-084fa14fbea4-must-gather-output\") pod \"e514d9d9-ed69-4be2-8161-084fa14fbea4\" (UID: \"e514d9d9-ed69-4be2-8161-084fa14fbea4\") "
Sep 09 21:09:09.580301 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.580261    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e514d9d9-ed69-4be2-8161-084fa14fbea4-kube-api-access-tmds8" (OuterVolumeSpecName: "kube-api-access-tmds8") pod "e514d9d9-ed69-4be2-8161-084fa14fbea4" (UID: "e514d9d9-ed69-4be2-8161-084fa14fbea4"). InnerVolumeSpecName "kube-api-access-tmds8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:09.585294 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.585270    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e514d9d9-ed69-4be2-8161-084fa14fbea4-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e514d9d9-ed69-4be2-8161-084fa14fbea4" (UID: "e514d9d9-ed69-4be2-8161-084fa14fbea4"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:09.679406 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.679376    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e514d9d9-ed69-4be2-8161-084fa14fbea4-must-gather-output\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:09:09.679406 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.679400    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmds8\" (UniqueName: \"kubernetes.io/projected/e514d9d9-ed69-4be2-8161-084fa14fbea4-kube-api-access-tmds8\") on node \"ip-10-0-138-141.ec2.internal\" DevicePath \"\""
Sep 09 21:09:09.896009 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:09.895972    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e514d9d9-ed69-4be2-8161-084fa14fbea4" path="/var/lib/kubelet/pods/e514d9d9-ed69-4be2-8161-084fa14fbea4/volumes"
Sep 09 21:09:10.413135 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:10.413099    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sx2kw_must-gather-jwks7_e514d9d9-ed69-4be2-8161-084fa14fbea4/copy/0.log"
Sep 09 21:09:10.413641 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:10.413617    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sx2kw/must-gather-jwks7"
Sep 09 21:09:10.413641 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:10.413628    2578 scope.go:117] "RemoveContainer" containerID="e6c2118feaa90c64ab2d28c46aed023083893bebed6925548af23c9cb3b2967b"
Sep 09 21:09:10.421472 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:10.421427    2578 scope.go:117] "RemoveContainer" containerID="0487ebf40279de3a9aca9d1fe6cf2c662811b6e61cfd942b3631d290909e24df"
Sep 09 21:09:10.480788 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:10.480757    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:10.583403 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:10.583371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jw9j8_c2b45f30-896c-434f-b5a5-029893e1fd1b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:12.953126 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:12.953087    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-c6hdh_816edc7c-ffa4-4cd5-af7d-4ed59ffa30e4/authorino/0.log"
Sep 09 21:09:13.054785 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:13.054751    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-9hp8f_cd920ac6-11e4-4c6a-99c1-d48bbf0500ba/kuadrant-console-plugin/0.log"
Sep 09 21:09:15.170009 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:15.169978    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wttd_7cfa4555-ad79-42c5-820f-71d642689608/node-exporter/0.log"
Sep 09 21:09:15.235643 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:15.235616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wttd_7cfa4555-ad79-42c5-820f-71d642689608/kube-rbac-proxy/0.log"
Sep 09 21:09:15.267149 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:15.267132    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wttd_7cfa4555-ad79-42c5-820f-71d642689608/init-textfile/0.log"
Sep 09 21:09:17.694998 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.694957    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"]
Sep 09 21:09:17.695423 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.695259    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e514d9d9-ed69-4be2-8161-084fa14fbea4" containerName="gather"
Sep 09 21:09:17.695423 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.695270    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e514d9d9-ed69-4be2-8161-084fa14fbea4" containerName="gather"
Sep 09 21:09:17.695423 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.695289    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e514d9d9-ed69-4be2-8161-084fa14fbea4" containerName="copy"
Sep 09 21:09:17.695423 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.695295    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e514d9d9-ed69-4be2-8161-084fa14fbea4" containerName="copy"
Sep 09 21:09:17.695423 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.695392    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e514d9d9-ed69-4be2-8161-084fa14fbea4" containerName="copy"
Sep 09 21:09:17.695423 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.695403    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e514d9d9-ed69-4be2-8161-084fa14fbea4" containerName="gather"
Sep 09 21:09:17.697290 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.697186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.701053 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.701033    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g4ljz\"/\"default-dockercfg-mjs97\""
Sep 09 21:09:17.701172 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.701036    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4ljz\"/\"openshift-service-ca.crt\""
Sep 09 21:09:17.701172 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.701073    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4ljz\"/\"kube-root-ca.crt\""
Sep 09 21:09:17.711682 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.711656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"]
Sep 09 21:09:17.740130 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.740089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-podres\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.740367 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.740157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgc9\" (UniqueName: \"kubernetes.io/projected/3bf015f8-6d75-43ea-9861-92a887403b72-kube-api-access-rmgc9\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.740367 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.740254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-lib-modules\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.740367 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.740294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-proc\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.740522 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.740374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-sys\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.840879 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.840838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-lib-modules\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.840879 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.840881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-proc\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.841158 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.840910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-sys\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.841158 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.840974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-sys\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.841158 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.840981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-proc\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.841158 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.841007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-podres\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.841158 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.841057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-lib-modules\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.841158 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.841063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgc9\" (UniqueName: \"kubernetes.io/projected/3bf015f8-6d75-43ea-9861-92a887403b72-kube-api-access-rmgc9\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.841158 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.841144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3bf015f8-6d75-43ea-9861-92a887403b72-podres\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:17.851131 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:17.851106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgc9\" (UniqueName: \"kubernetes.io/projected/3bf015f8-6d75-43ea-9861-92a887403b72-kube-api-access-rmgc9\") pod \"perf-node-gather-daemonset-687qj\" (UID: \"3bf015f8-6d75-43ea-9861-92a887403b72\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:18.007515 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:18.007415    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:18.281000 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:18.280901    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"]
Sep 09 21:09:18.284267 ip-10-0-138-141 kubenswrapper[2578]: W0909 21:09:18.284236    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3bf015f8_6d75_43ea_9861_92a887403b72.slice/crio-ac6170173eee20387e8c2604af496013fdabae8580e0e93f60c9c79cf1286f53 WatchSource:0}: Error finding container ac6170173eee20387e8c2604af496013fdabae8580e0e93f60c9c79cf1286f53: Status 404 returned error can't find the container with id ac6170173eee20387e8c2604af496013fdabae8580e0e93f60c9c79cf1286f53
Sep 09 21:09:18.456639 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:18.456600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj" event={"ID":"3bf015f8-6d75-43ea-9861-92a887403b72","Type":"ContainerStarted","Data":"da8c7d339a0a0deafa4d8ad283b4cbc4909e06ebe02c4d0a621986aba094483e"}
Sep 09 21:09:18.456828 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:18.456661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj" event={"ID":"3bf015f8-6d75-43ea-9861-92a887403b72","Type":"ContainerStarted","Data":"ac6170173eee20387e8c2604af496013fdabae8580e0e93f60c9c79cf1286f53"}
Sep 09 21:09:18.456828 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:18.456687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:18.475652 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:18.475581    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj" podStartSLOduration=1.475561382 podStartE2EDuration="1.475561382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:18.474352485 +0000 UTC m=+4467.063006138" watchObservedRunningTime="2026-09-09 21:09:18.475561382 +0000 UTC m=+4467.064215044"
Sep 09 21:09:19.687820 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:19.687790    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-fm5k6_08e820ae-bca1-46fb-be7e-ef43f0fefdd5/volume-data-source-validator/0.log"
Sep 09 21:09:20.777320 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:20.777277    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82kmf_eb6371e1-d79c-449d-a6a6-1db356e8fdf5/dns/0.log"
Sep 09 21:09:20.848398 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:20.848367    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82kmf_eb6371e1-d79c-449d-a6a6-1db356e8fdf5/kube-rbac-proxy/0.log"
Sep 09 21:09:21.143721 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:21.143683    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ztrxp_cc50338f-4fdf-41c7-a40e-19d0e3ebb9c6/dns-node-resolver/0.log"
Sep 09 21:09:21.863561 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:21.863526    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rxfgd_175a2b7e-f1c9-4a5c-bc4c-75f73cc79808/node-ca/0.log"
Sep 09 21:09:23.030278 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:23.030234    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/0.log"
Sep 09 21:09:23.044187 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:23.044158    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/1.log"
Sep 09 21:09:23.798364 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:23.798327    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5x4l5_65266674-4ef3-4abf-b20c-3c06447af305/serve-healthcheck-canary/0.log"
Sep 09 21:09:24.314520 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:24.314482    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-s2v7q_8fcff86d-6410-409d-975b-877aa305ceb1/insights-operator/0.log"
Sep 09 21:09:24.315026 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:24.314985    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-s2v7q_8fcff86d-6410-409d-975b-877aa305ceb1/insights-operator/1.log"
Sep 09 21:09:24.471253 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:24.471222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-687qj"
Sep 09 21:09:24.563659 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:24.563632    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mzd2f_aadad933-4edc-411c-9787-38eea928a954/kube-rbac-proxy/0.log"
Sep 09 21:09:24.597852 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:24.597826    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mzd2f_aadad933-4edc-411c-9787-38eea928a954/exporter/0.log"
Sep 09 21:09:24.638548 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:24.638521    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mzd2f_aadad933-4edc-411c-9787-38eea928a954/extractor/0.log"
Sep 09 21:09:28.191360 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:28.191328    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7ff45979b5-r9jv5_5e80ba5d-4e55-47d6-b516-0dfad1d7223b/manager/0.log"
Sep 09 21:09:28.520265 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:28.520224    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/0.log"
Sep 09 21:09:28.534708 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:28.534682    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/1.log"
Sep 09 21:09:29.278063 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:29.278033    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-z4tn9_5bce6bde-3815-41cf-916c-69f883449e09/server/0.log"
Sep 09 21:09:29.801438 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:29.801345    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/0.log"
Sep 09 21:09:29.819741 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:29.819707    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tbtbv_870011d5-3b7b-4211-8007-9f917ffa1944/istio-proxy/1.log"
Sep 09 21:09:30.087516 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:30.087428    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-v98qv_d43d0471-02f4-4bcb-ba24-9f055ac3636b/manager/0.log"
Sep 09 21:09:30.245139 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:30.245104    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-98mwf_bf2f825a-3ff3-4a47-9b7e-1ffa22eec41b/seaweedfs/0.log"
Sep 09 21:09:35.129518 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:35.129487    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fspqg_c55bb2b8-6bb7-436e-86e2-f517432078cb/migrator/0.log"
Sep 09 21:09:35.163291 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:35.163264    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fspqg_c55bb2b8-6bb7-436e-86e2-f517432078cb/graceful-termination/0.log"
Sep 09 21:09:35.616914 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:35.616885    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-9ts9g_82c99ae4-950c-4770-bd66-d8778fe3baf4/kube-storage-version-migrator-operator/1.log"
Sep 09 21:09:35.617913 ip-10-0-138-141 kubenswrapper[2578]: I0909 21:09:35.617869    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-9ts9g_82c99ae4-950c-4770-bd66-d8778fe3baf4/kube-storage-version-migrator-operator/0.log"